First of all, don't panic.
Secondly, you probably SHOULD contact a RE Lawyer because from your postings it now sounds like you entered into a DUAL AGENCY with the Seller/Listing Agent, which in my opinion is a "Double Whammy"!
Your Buyer/Seller/Listing Agent is most likely quite familiar with the CAR Residential Purchase Contract - which functionally and reasonably leaves you without the capability of advantageous representation or guidance. I'm confident a RE Lawyer would be very interested in your "representation scenario".
[Note: A DRE licensee in CA can simultaneously act as a Buyer and Seller's Agent in a transaction - provided consent of both Buyer and Seller is documented. In fact, an undisclosed Dual-Agency is a felony in CA.]
Going back in time, had you secured your own Buyer's Agent at this point they might also be informing you to seek legal counsel in addition to making you aware of the following CAR Residential Purchase Contract sections; which you may be interested in reading if you have not done so already:
1) The Seller quote you provided is from Para 10(A)(v) of the CAR RPA. The first sentence preceding what was quoted refers to the Buyer satisfying matters contained in the Buyer's Inspection Advisory. I suspect you might be interested in reading what is listed in Para 12 of the Buyer's Inspection Advisory.
2) Contract contingency removals in the CAR RPA are not "passive" but instead require "active" removal. I would suggest you comprehensively read Para 14 A, B, & C (pay close attention to 14C3). Finally, study 14F regarding the release of deposit funds!
3) Note Para 26A & B provides for Mediation; and if no solution is arrived at, Binding arbitration; however, Para 26C(1) excludes these as a requirement if the amount sought is within small claims court jurisdiction.
Best Regards, Steve