The standard California purchase contract specifies a 17-day contingency period, so I'm surprised that you were allowed 21 days.
The California contracts were changed recently to favor the seller in most situations. I'm assuming that you signed a Contingency Removal form rather than a Counter Offer, which clearly states that "Buyer's deposit shall not be returned if Buyer does not close escrow. This could happen even if, for example, Buyer does not approve of some aspect of the Property or lender does not approve Buyer's loan."
Your Realtor should have explained contingency periods to you at the time the purchase contract was signed and at the time you removed your contingencies, and of course someone who wasn't present when you signed these contracts cannot be sure what was said (or not said) during these conversations.
The contingency periods are meant to protect the seller, since tying up a home in escrow means that the sellers cannot continue to accept offers from other buyers. Backing out either because you changed your mind or couldn't get the desired loan approved is not considered to be a valid reason for cancelling escrow once contingencies have been removed, so you'll likely have a tough time getting a resolution through the legal system. The legal department at the California Association of Realtors is very clear on this point - there is no obligation for the seller to return the deposit to the buyer in this situation, nor is there any obligation for the Realtor to refund the buyer's deposit out of their own funds.
You should talk with the Realtor's Broker and see if there can be any sort of voluntary resolution where you'll get a portion of your deposit back from your Realtor, although they are not obligated to do this.