Your situation is a classic example why it very important for the buyer to chose wisely who will represent him or her when they enter the real estate market. The business environment today calls for experience, and absent that, education and training. The blind can't lead the blind unless they both fall into a ditch. I wish you the best.
I've seen lenders do what you're talking about over and over...it's not uncommon. Have your lender and agent contact the listing agent...and try to work out a compromise.
Alissa Spears http://www.HomesInRockport.com
But typically, either party can back out... if the milestones written into the contract are not met. Your agent would be knowledgeable in this... if your agent is telling you that it's allowable.... then it's probably allowable.