Interesting proposal. I offer two thoughts to you. First, consider a modular home. I know, I know...they are manufactured homes. But, the designs are WAY better than they used to be. (I have a picture of one a friend is going to build, if you'd like to see what's possible today.) Additionally, these homes are built inside a climate-controlled factory with strict quality controls. No weather issues. With all due respect to my friends in the building industry, I don't know why more homes, or at least parts of homes, aren't built this way.
Secondly, there are companies who move homes. They section them up and put them on wide-bed trucks and ship them then put them back together. Mostly used for historical structures, but certainly possible for the average person. It only takes money. :)