You need a lawyer to review the details of your contract. If there were no contingencies, such as building inspections or mortgage approvals, the buyer would be hard pressed to walk away without loosing his right to a return of his deposit. This is why a sizeable deposit is desired. Rich
One of the possible consequences aside from those already mentioned is the fact that your agent and/or the agent for the buyer, brought a ready, willing, and able buyer to the table. Each could demand payment of the agreed upon commission in your signed agreement.
If you are in an executed contract--signed by all parties--and deposit money was exchanged--do consult with an attorney who specializes in real estate, have all documentation reviewed and inform yourself regarding all ramifications that may/will occur. Your attorney will best advise.