Your lawyer is the one who has access to the contracts that control your transaction and the ability to ask you direct questions and get specific answers. We would be practicing law and our psychic abilities to provide a more qualified answer.
When there is a dispute between what an agent/broker says and what your qualified attorney says, I'd go with the attorney most of the time. If you are still uncertain, get another attorney's perspective.
Finally and most importantly; why the change of heart? You are still upside down on you unit, you endured months of preparation, showings and waiting for a bank to respond having put a lot of people to work and now you have a change of heart?
Your agent is only compensated for a closed transaction; of course they are upset over working and investing time and money into your deal and receiving nothing out of it. The buyer's agent is in the same position; the buyer's have bypassed other homes they could have purchased only to be let down.
I'm not saying you should complete the sale to be "nice' to the brokers and the buyer against your interests, but you should consider carefully all that you've gone through and asked others to before you just change your mind.