You mention his Realtor, do you have a Realtor representing you? There are so many great homes out there for sale. Do you really want to put yourself through all the stress and perhaps long legal battle?
Pehaps if you try to negotiate to have the seller reimburse you for your expenses ie appraisal if completed and inspections costs in lieu of taking him to court he might agree. That way you can move on and find a new home. In any case you would be entitled to the return of your good faith deposit. Good luck!
Century 21 Longacre Realty
Bally, Pa. Berks Co.
610-845-6000 x 208 cell 610-442-6897
Depending on the language of your purchase agreement, you might be able to keep the buyers earnest money as remediation of damages. I'm not an attorney, but I'm guessing your attorney is trying to help you reach some sort of settlement without using a lawyer as the attorneys fees may well outstrip any monetary gain.