Was this a FSBO?
Who does your purchase agreement state is holding the money?
Is the deposit amount included on your HUD statement?
Why is this becoming an issue 3 years after the fact?
The HUD is an account of all funds involved in a transaction that should clarify the presence of your deposite money relative to this transaction.
I find it difficult to believe that this could happen when involving licensed agents and professional closing personnel....but then anything is possible!