There are, in some locations, "voluntary home owner associations." Owners are not mandated to join, or pay fees. There are weakly enforced rules that exist in the best interest of the community but there is very little in the form of enforcement.
Your situation sounds like it could be one of these arrangements.
If this is the case, we suggest requesting copies of the association rules and by laws so you will know what guidelines you are NOT following....
You are right that you should already know if you're in an HOA if their rules are still valid. It would have, or at least should have, been disclosed by the seller in the purchase contract (if you used a Realtor and thus the California Residential Purchase Agreement). You should also have received additional documentation from the Association during the escrow period.
If you still have your paperwork, check the Preliminary Report of Title. There would be a reference to CC&Rs (Covenants, Conditions, and Restrictions) that govern the property in the list of excluded items. A title/escrow company would normally include copies of this documentation with the Prelim as a courtesy to the seller.
Do you pay monthly or yearly dues to an Association? If so, you are probably in some sort of an Association, but its ability to govern what you can or can't do to your property may be limited.
In Santa Barbara and Goleta there are a few areas that were at one time governed by CC&Rs, but I don't believe they're still actively followed. For instance, just outside of Hope Ranch, there's an area referred to as the Hope Ranch Annex. This area has CC&Rs from the 1920s or 1930s. There are also areas near La Colina Jr High School, off of Turnpike Rd, off of old San Marcos Rd , and near Los Carneros Rd that share a common pool and clubhouse. There may be community fees paid in these areas, but I believe all residences are treated as single-family homes. Any title restrictions or CC&Rs would likely focus on pool dues and use instead of guidelines regarding remodeling, construction, painting, yard maintenance, trash cans, street parking, etc, of your own residence that are often found in "typical" housing developments.
Check your records, call your Realtor, and/or ask the complaining party what documentation they have that makes them believe you're subject to CC&Rs, and if so, what stated rules you're violating.
Best of luck,