The first thing to look at here is your credit score. When buying an investment property, you must use conventional underwriting (Fannie Mae or Freddie Mac). Both companies require at least a 680 credit score however, realistically, the bank won't approve you without a 700 score. Guidelines afterall are meant to be changed. So, if your score was below 700, this deal was already dead.
Next is the HVCC factor. The Home Valuation Code of Conduct law requires that no one talk to the appraiser before or during the appraisal appointment. This law was passed to keep LO's and other real estate entitities from harrassing the Appraiser into giving properties higher values than they supposed to be. Stated as one of the reasons the real estate market crashed (whatever). The problem with HVCC is that the appraiser gets paid through a management company at half of what he/she is supposed to get so the appraiser doesn't spend as much time on the property looking for better value because he/she has to get back into the hopper. The Real Estate Association among others are trying to kill this law as it's killing the value of properties around the country.
The next thing is the condo market. Investment properties require at least 20% down (unless going through HomePath). Depending on your score and the condo questionnaire, the lender may require an extra 5% down from the buyer. You stated that you were putting 25% down and paying 4 points. This may be a sumation of 1.75 points for investment property. The other points may have been because it's a condo and/or because the loan size is under a certain amount (every lender has different point systems).
I would recommend you look into the HomePath program. There are a lot of foreclosures at good prices and currently fannie mae is paying 3.5% in closing costs through December 31, 2010. The minimum down is 10% and the minimum score is 680. No appraisal is required and there is no mortgage insurance. I've added the website below.
If you start this process over again quickly, and make sure that this offer is coming from your attorney with HIS cover sheet, I think this broker will make certain he will not play games or otherwise misquote you, try to get you to buy another appraisal or put his hand into your pocket.
Scott Miller, Realty Associates, Boca Raton, FL
You can find my contact info on my Trulia page if you would like some help.