What is the name of the community? I believe you just got yourself a leverage. Give me a call, lets work on a strategy.
You correctly note that wasn't a contract. Your agent should have thoroughly briefed you on what this was and made it clear. As for the agent being out of town - well that seems to have cost you want you wanted. You and the agent should have discussed your desire for this lot and the contract should have been written.
At this point either hope this falls apart and you get another shot at it or move on to a new lot. You might also want to look into an agent that's better able to meet a client's needs - this is a business where things are lost very quickly and unexpectedly. There is no excuse for an agent being "out of town" without back up.