I'm sorry to hear about your situation. My recommendation would be to contact an attorney that specializes in Real Estate Law first thing Monday. Disclosure laws in California are simple. You must disclose all known Material Facts regarding a home when you sell the property. A partial disclosure by the seller(s) will not cut it. A buyers responsibilty to complete their Due Diligence on a prospective property does not take away a sellers responsibility to disclose all known material facts. A good rule of thumb that I always give my clients is WHEN IN DOUBT, DISCLOSE! Best of luck!
You knew about the unpermitted modifications and didn't regard the risks. A simple permit search would have revealed outstanding permits. I just recently talked to a lady in a similar situation. Her neighbors were not friendly and reported her illegal modifications to code enforcement. That could have happened to you as well.
I am truly sorry for your predicament. The seller made prudent disclosures and the California Real Estate forms are a large pile of "Buyer Beware."
This is a really unfortunate situation for you. Based on the info that you have provided, I do not think the issue can be settled by any opinions you get on this site. My advice is to talk to a qualified real estate attorney as soon as possible to determine your recourse. Best of luck to you.
The lesson for all other buyers out there is to investigate all issues to their satisfaction during the property contingency phase prior to completing the purchase. They should also be sure to work with a competent Realtor to get complete disclosure and satisfactory advice.
Take care and good luck,