Good answers below. I will add that your question doesn't give us complete information. What do you mean that the "deal fell through?" That sounds like the transaction is not going to happen, but the question is "why?" Was there a breach of the contract? Or, were the contingencies not met per terms of contract?
The answers from the other posters really apply to if there was no breach of contract. If there was no breach, both parties must sign a Return of Escrow Deposit form before the earnest money can be returned. As noted by other posters, if there is disagreement on the return, then it must be reported.
If there was a breach of contract on your part, the earnest money may be taken as damages by the other party.
Realtors should avoid "hanging on to deposits" as that is illegal. Brokers can hold deposits in escrow account, but have no reason to do so since title companies do it. The number one violation of brokers is inaccurate reporting/handling of escrow deposits. My broker doesn't have escrow accounts. We have title companies hold all deposits.