Not knowing anything of your particulars, Let me assume that you do not owe anything for your land, if you build a "stick built" home VS. a Modular home.
Pretty much everyone is familiar with the standard house, built on site, mostly by hand, from basic materials. Called "stick built" to differentiate it from other building methods, this is the default housing that everyone is familiar with. Once emplaced upon that property, there is no real way of getting it off the property intact, and therefore it is appurtenant to the land. This might come as a shock to people who concentrate on the house, but when you buy a property, you are buying the land upon which it sits - the lot - and the structure comes along because it is appurtenant - attached and cannot be moved off easily. It is this type of property which has been at the forefront of liberalization of lenders loan policies, precisely because it is both universally desirable and non-portable. That land is defined by its boundaries. It isn't going anywhere. The structure isn't going anywhere that the land isn't, because in order to remove it, you pretty much have to destroy it. It's built on a several ton concrete foundation, which, if you nonetheless manage to pick it up, is still overwhelmingly likely to crack if not disintegrate, not to mention ripping out plumbing, electrical, and other connections.
Now because the land isn't movable and the structure isn't either, lenders have gotten comfortable that you're not going anywhere with that structure. Because the combination is so universally desired among consumers of housing, they have gotten comfortable with giving loans for almost the full purchase cost of the property (and I will bet you money 100% financing will be available again on some terms within 5 years), knowing that it takes a special set of circumstances for them to take a loss on the property, and they can charge higher interest rates in order to insure against that. (I am using insure in the statistical, law of large numbers sense that is the essence of insurance.)
The next step away is manufactured housing on land owned by the home owner. Now technically speaking, modular housing is a subset of manufactured housing, but when most lenders are talking about manufactured housing, they are talking about homes built at the factory in entire sections, and assembled with only a few total joins at the home site. True manufactured housing is portable, where modular really is not. If you're in Idaho and decide to move that house to your property in Georgia, it's doable.
Now because it is portable, as you might guess from things I've said here about the prevalence of attempted scams that lenders have had issues with people dragging them off. You'd be right. Lenders file foreclosure papers on the land, and the homeowner metaphorically backs up the pick-up truck and takes that residence somewhere else, leaving the lenders with a piece of land and no residence. Because there is no longer a residence on it, it's not worth anything like what it was when there was a residence on it. Lenders have lost multiple hundreds of thousands of dollars on individual properties around here. You get burned enough times, you start getting wise. Those real estate lenders who will lend on manufactured homes require a laundry list of conditions, and even if they are all met, they won't loan 100 percent of the value, or anything like it, and there will be an additional charge of at least one full point of cost on their regular loan quotes. Cash out loans are typically limited to sixty-five percent of value, making it hard to tap equity. Furthermore, due to accounting standards and depreciation, Fannie Mae and Freddie Mac made a rule that manufactured homes were limited to twenty year loans, which drastically limits not only the type of loans available to their owners, but also has the effect of restricting what they can afford to borrow, because the payments principal has to be paid back over a shorter period, and the difference between a twenty and thirty year repayment is much greater than the difference between thirty and forty.
Lest anyone think that this is in any way shape or form due to inferior construction, it is not. Because these buildings are manufactured on assembly lines which are largely robotic, there are many fewer problems with things like forgetting to nail at appropriate intervals, workers getting distracted, not getting corners square, and all those sorts of problems. I'd bet that a manufactured dwelling is probably of superior construction to a site built dwelling, all other things being equal. It is purely lender policy, as influenced by the history of their experiences with these kinds of properties, which is driving these differences.
All that being said, don't rule this option out as i myself am very happywith it. AND, NO, My house does not look like a MOBILE HOME!