You should be able to withdraw your listing and not owe your realtor anything. The realtor usually only gets paid when the sale goes through. Quite often things don't work out for one reason or another and that is part of the real estate business. Real estate professionals are used to this and should handle it for you with no problem. If the agent gives you any problems, you can always speak with the broker of the agency they work for. As for the buyers, their offers were always contingent on the bank's acceptance which you never received so you should be able to cancel. If you never signed the purchase contract you were never in a contract to begin with. Good luck to you and I hope you can keep your home.
Since you did not accept any of the offers you are not obligated to sell your property. Call your agent and explain the situation, meaning, that you are no longer considering a short sale and looking to modify and stay in the property. Good agent will understand and hope you'll call them back when you decide to sell in the future.
Your isting agreement will specify the terms of your agreement with your listing agent, but in most cases (if it's the standard agreement) your agent is compensated at the time of closing. While it would be nice to compensate them for any expense that they incurred in attempting to sell your house, I don't believe you are obligated under the contract.
Listing agreements expire within a specified period, normally 3-6 months, and then is renewed if no sale has occurred by then. Others have an 'easy exit' clause where you can cancel at any time if you are not satisfied wtih a 24 hour written notice. But that is to protect the agent, who has incurred expenses in the process of marketing your home, from being pushed out and you selling your home on your own or with another agent. Most would be willing to cancel the agreement should you decide to stay in your home.
Read the contract. That will give you your answer.
I would review your contract with your listing agent. Then give your listing agent a call and explain the situation. Most agents are understanding and know that things can change. They will want to call the buyers and let them know that you are probably going to take your home off the market. The chances are good that the buyers have been continuing to look for other homes anyway. In this market short sales are not a sure thing.
It is advisable that you speak with your agent and discuss the listing agreement. Given the fact that the bank may take over 60 days to respond to the offers and you have second thoughts about selling your home, your agent should be willing to let you out of your contract.