Very good question and one that I am dealing with, in a different form, myself with my buyer client. Let's speak to what typically (although that is not a true description of what tends to happen in the world of real estate since every case is different) happens. The builder makes a request of a deposit to get the building started. Because of the risk that the builder is taking, the deposit is usually written in as being non-refundable and goes directly to the builder so s/he has funds to begin the work. This money is seen as being used on your behalf as long as you perform according to contract. If you fail to buy the finished product, then the builder's loss is reduced. Mind you, this has nothing to do with whether the builder did a good job or not in building the home. That is a separate issue.
That 5% deposit is money that is to be used by you in the manner that benefits you the most. It will, many times, include your 3.5% down payment as well as money for your closing costs. If you are getting your closing costs covered by the builder, then you could very well end up with a check from the title company for the difference after closing.
If you have any other questions or concerns I would suggest speaking with your buyer's agent or whomever wrote up the contract to begin with and asking that person questions regarding your particular situation. Your loan officer will also be a very good resource.
As always, you can also feel free to contact me and ask your questions. There is no such thing as a stupid question with me...and, I am happy to say, with most of my fellow agents.
Ed Rippee, CSP, WHS, CDPE