This is predicated upon the assumption that the seller side has no dispute. If the seller side disputes your claim and is unwilling to acknowledge the buyer's right to terminate the contract, the monies will be held while the dispute is resolved.
Whoever is holding the funds must have had the same deposited in their account for a sufficient time period for clearing. Some firms (attorney, brokerage or title) will have policies requiring up to 10 days from the date of deposit for this.
The deposit cannot be released until both attorneys (assuming each side has counsel) signs off on the release. If I, as a broker, were holding a deposit, I cannot release the deposit until both attorneys sign off on the release. Attorneys might sign off in a day, or take a week or longer. Once releases are signed, the firm holding the deposit can then process the refund through their system, and processing times vary for different firms.