I would like to point out that the standard contract used in California contains verbiage in paragraph 14.F. that may be pertinent to your situation.
Legally, you should consult an attorney about any problems with a contract. Believe it or not, real estate agents & brokers are not legally qualified to discuss the legalities of any contract.
In my (non-leagal advice) opinion, they seller cannot hold your deposit and require your performance on the contract without cause and could be subject to fines for not doing so. Ask your agent for a copy of the cancellation paperwork and it will likely say something to that effect on it - then walk it over to an attorney or contact the office broker to lean on the other agent.
Hope that helps.