I'm no lawyer and you certainly need a real estate lawyer here because the broker is not cooperative. Remember the broker is legally responsible for the action of the agent.
The fact that the same agent represented you and the seller is extremely unusual and requires special cautions which appear not have been taken in your case. In my opinion if the facts are as you describe you are entitled to not only your deposit but damages as well. Whether that can be achieved at a reasonable cost only an attorney can tell you. Even if you do win a judgment, collecting on it may not be possible if the parties are gone (you said the title company is closing) or don't have any ability to pay or assets to seize. If that's the case it is probably fruitless to sue, but again that's something on which you need a lawyer's advice (and one that has experience in real estate law). That said, let me try to clarify some issues.
Was your offer made on the standard form provided by the California Association of Realtors? (I realize that may be a hard for you to answer since you appear not to have a copy of your documents.) It's titled "Residential Purchase Agreement. If that form was used, there is a optional provision at the bottom of page five allowing the parties (buyer & seller) to agree to arbitration. If both parties initialed that paragraph you cannot sue, but must go to arbitration (you still need a lawyer). That may be to your advantage.
As for you owing any money to the seller as compensation for the length of time it has taken, that would not be the case unless you signed a document agreeing to it. There is no standard document for that (nor have I ever heard of any seller asking for that) but it may have been included in a form called an "Addendum". That form allows one to write free-form any conditions either party wants to add to the contract. However no form is legally binding unless both parties have agreed to it.
As for the concept of a buyer paying a seller for delay, it could reasonably come in the form of an extension of the contract expiration date. It might occur that a seller would be unwilling to extend a contract without the buyer paying something for the privilege. Highly unusual. In my experience sellers either agree or not, without any agreement to additional money. If I read you correctly your seller made such a request but you did not sign the agreement so you are not obligated to pay it.
There is a standard form and process through which a seller can give notice to a buyer (or vice versa) that unless they do what is said in the contract (usually agree to buy the property) within three days of the notice, the contract is void. In principle, if your seller was concerned about the amount of time it was taking to close your deal, they could have sent this notice and, unless you performed or both parties agreed to something else, the seller would have been free of your contract and normally would have to return your deposit, but that's a stickier matter resting on more detail than we have here (and more legal experience that I have). Again, this is a matter for a real estate lawyer.
Whether a buyer is entitled to a refund of their deposit depends on several things, the most important of which is whether they have signed the document(s) removing all contingencies and it seems unclear to you whether you did that or not.
I hope that provides a little clarity and I wish I could provide reliable advice, but if there was ever a case which needed a lawyer this is it. I wish you the best.