First I would be sure that it is illegal. It is very hard to tell. When you go down to the city might find something saying, new sink permitted this is actually pointing to a set of blue prints, it was is on the blueprints that is permitted.
I had one that did just that. The label on the permit said a kitchen sink. it was actually a complete kitchen remodel, a spiral staircase on the outside of the home which led to the roof which had a deck on it and a room full addition. I was the listing agent and the buyers agent was trying to back out saying none of it was permitted. Luckily I found the Blueprints in the attic and could then prove that they improvements were permitted.
I used to be a licensed Dept of Consumer Affairs home inspector (Branch 2) and if the home was built to code, I would pass it. Really, I have seen several of these things were people are sure the improvements were not permitted when in fact they were.
Don't get me wrong, people add on illegal additions all of the time in order to avoid paying additional property taxes. Truly you have to find the blue prints in order to know if these are permitted or not. So the best one can do, because the blueprints are copyrighted by the Architects is to assume that if it was built to code it was probably permitted.
I would the the seller is liable, if they knew these additions were not permitted. The city can make your remove the improvements which can cost you money doing the work as well as lower the value of your home. If the agents knew the work was not permitted they would be liable as well.
Assuming that no one could tell the structures were not permitted tells me the property was probably built to code. Get the city out there and try to get it permitted.
Realize though, that you are waving a red flag at a bull. The City inspector could decide that the improvements are not to code and make you tear them down. Leaving you having to go back and try to get someone else to be liable for the expense.
If the property were a rental or foreclosure before it was sold, California law pretty much says that the purchase is an as is affair and you may not be able to collect.
Be proactive as well as you have a limited time to bring it to court.