Joe the builder hit on some of the most expensive and time considerations many people don't factor in. You'll also have to consider the financing side of it. If you have saved enough cash to buy the land outright you'll be way ahead of the game and make it less expensive and easier. Since you listed 23185 as the zip code I assume you are looking to do this York or James City County. Both will allow pre-fab construction but there will be some initial permit and development issues you'll want to talk with zoning and code compliance about before you begin buyng or spending any money. I think land should be your first and most important consideration, then speak with the county.