Please refer to your contract, terms, duties and obligations. By definition, your deposit might be forefeited if backing out of a deal after final contingency removal. California contracts have a liquidated damages clause that limit the amount of damages to 3% of the buyer deposit. If you and the seller did not agree to liquidated damages, then the entire deposit is up for grabs.
Reiterating what the other agent stated, if you are not acting in good faith, for example, you found out that a bank approval came through on a property that you wanted before getting into contract on the current property and need the deposit to open a new escrow. OR, you found out that your loan terms would not be as favorable as you thought but still within the financing terms set forth on the offer....so on and so forth. In these cases, the seller can try to seek remedy for lost marketing time if it resulted in a loss of value on appraisal, or something like that, etc.
The situation could require a mediation panel to make a ruling that is binding, after hearing both parties and the circumstances if the seller is refusing to willingly sign off and release the funds.
Note: Clarification on contingencies, merely 'passing' the contingency date does not constitute automatic removal; buyers must notifiy the seller in writing of such or cancel the contract.