There are rules, there are legal opinions and legal reality, but in the end there is the set of facts. Regardless of the rules and what you state in your contract, the question any good buyer's attorney is going to explore is whether you have sufficient knowledge and experience to have known that the buyer could not buy the home using an FHA loan. The buyer's attorney is going to try to prove you are a sophisticated business person who knows enough about flipping homes to have known that the FHA anti-flipping provisions would prevent any buyer from using an FHA loan. If he succeeds, no court, whether jury or judge is going to look kindly upon you. Instead, it will look at you as a predator who took money from a buyer when you knew he could not compete the deal. In short, they'll see you as a thief taking advantage of a ignorant person to steal his hard earned money and prevent him from being able to own a home. Do you really want to maintain your position when you are faced with such a probable outcome?
Now, your statements in your question are basically admissions that you know the legal concept of "constructive notice" and you relied upon that to protect you from having to disclose that you had the property for too short a time for an FHA loan to be able to be used for the property. It appears from your statements that you had knowledge of the material fact that you had too short a time in the home to allow an FHA loan to succeed in the purchase. (Your agent certainly should have had such knowledge and had a duty to advise you of such result. If you are a realtor, you have a duty to know such things and disclose them to the buyer and seller.) Constructive notice is no substitute for disclosure of a material fact within your personal knowledge when holding out a home for sale as a principle dwelling in California. By accepting an FHA loan you are stating you know that the home is going to be used as a principle dwelling. Also, the contract itself most probably probably stated on it that the buyer was intending to occupy the dwelling as his principle residence.
By misleading a buyer that he can purchase a home with an FHA loan and accepting his deposit when you know you have flipped the home within too short a time for an FHA loan to be able to be used, and relying upon the sales contract to keep his money sure looks like you knew that you, in the end would keep is money and keep the house. It looks like your only intention was not to sell the house but keep his money. Don't you think that makes you look like a thief by committing fraud? Are you aware of the civil penalties for fraud and criminal penalties for fraud? Well, stay your course, keep the money and you may find out!
Sure, you may "win" or you may not, but why take the risk? The legal fees necessary to defend yourself and prove your point, if you actually have a winning point will be far more than the deposit.
Words to the wise,