You need to talk to your own attorney ASAP. Either the lender's attorney you hired for yourself or a private real estate attorney to represent your own interests. Typically the lenders attorney is also in line with your interests but in a case like this there may be a difference in interests. Take any documents you've signed, any correspondences & any documents handed to you so far with the agent/broker/seller/attorney's and have everything reviewed by a qualified real estate attorney to review your options. Here in Massachusetts, I've never heard of a buyer "taking off with" a deposit without the buyer having signed legal documents allowing them to.
This is why you need legal representation to review what documents you have and what your legal options are. Check back with us to let us know how you made out, it's informative for others who read these posts,
Good luck with your efforts,