Unless we're talking over 50 miles away, and your currently have a home, that in of itself should not have been grounds to deny your loan. Based on what you shared by your credit score be good at one time, that sounds like the actual reason. Also, very few lenders are doing modular homes any more as well. Maybe your window of opportunity closed on that as well.
Talk to your mortgage rep - that's whose advice you need most. Ask for clarification and whether there is anything you can do to change the outcome.
Good luck to you,
Unwavering Commitment to Service
Do you own any other homes?