I'd say it depends on the age and quality of construction. If it's a more recent building, the quality of construction should be at least as good, if not better, than a site built home. Most of the modern modular homes are very well built and there should be no concern. There will be buyers out there who flinch at the term "modular", but they may just not be familiar with modern modular construction and are assuming it's a "manufactured home". There are a lot of different types of modular construction as well. some are stick built, some are panelized and built of styrofoam sandwiched between layers of plywood and there are many other forms of modular construction.
We have quite a few modular homes built in my community and they are higher-end homes where there is no difference in pricing because they are modular vs. stick built. I've been told by a couple of modular contractors that the materials used are of higher quality than what some builders use. It is true that modular homes go through much more rigorous inspections than a home that is not.
Modular homes are nothing like they used to be back when they first came onto the market as these modular contractors know that they need to compete with stick built homes and overcome the previous stigma of what the term "modular" used to be in the past. They can assure quality construction where some regular contractors may possibly cut corners. This usually satisfied the potential buyers about the quality of construction.
I hope this is helpful.
Prudential Connecticut Realty