You mention escrow, so presumably you put your deposit with a title company. If that's the case, the way it works is this:
Buyer or buyer's agent notifies the title company they want to cancel the escrow and get their deposit back. The title company then sends out cancellation instructions to buyer and seller for signature. If/when both parties sign the title co can release the funds. The title co is a neutral third party who will not adjudicate a dispute, so if the seller refuses to sign you will have to use the dispute resolution as outlined in your purchase contract.
If you follow up with more details I might be able to offer more specific advice.
Lance King/Owner-Managing Broker