If you just purchased now is the time to do it.
They already made the assessment of your home or soon will.
Contact the Riverside County Tax assessors office asap and let them know.
This may effect other things you do not expect to down the road if you don't.
Harold Sharpe - Broker
So Cal Homes Realty
California Department of Real Estate Broker License # 01312992
Also, as Sarah pointed out, sometimes it has to do with the septic (if you have one) and the lot size. Those things can influence how a house is classified and it may not be able to be changed if it doesn't fit the classifications for a 3 bedroom home in your area.
It could also be worth looking into permits that were taking out on your home, as previously pointed out. But if the appraiser was doing his job correctly (as it seems like he was because he picked up on the discrepancy) then he should have already looked into permits and such in the public records. But it wouldn't hurt to do your own research!
Quite often with homes this can be the case. I personally have a three bedroom house with an 'office' that we use as a fourth bedroom. But, we have a three bedroom home, septic, etc...and it is not possible to change it in my case.