The other advice below, about using a Realtor rather than an auction, is good...at least in about 95% of the cases. There are instances in which an auction may be the better choice, but not too often. (OK, I can hear the question: So, when is it better? It's better in a slow market...it's better if you can and are willing to accept something (maybe 10%) under what it would sell for listed with a Realtor...it's better if you're facing some time constraint and really have to sell in 30 days, versus 100-180 days or more...)
One other comment before we plunge into auctions: You shouldn't have to "get stuck" in a 6 month listing agreement. Demand less--say 3 months. And tell the agent: "If I think you're doing a good job, I'll be glad to extend my listing with you." And mean it.
So, auctions. There are several companies that I know of who do real estate auctions. It's a specialized field; a traditional auction company, as good as it may be, may not have the skills to handle a real estate auction. Here's a link to one I'm familiar with: http://www.auctionbrokers.net/
As with other auctions, there are two basic types: reserve and no reserve. Reserve means you set a base price. You retain the right to reject any offers below that price. In a no reserve auction, you agree to sell to the highest bidder, regardless of how low that bid might be. So, if your property might be worth (based on comps) $200,000, you might set the reserve price around $175,000. Could be higher or lower. So you could turn down a bid for $170,000. You'd have to accept a bid of $176,000. In a no reserve auction, if the highest bid was $140,000, you'd have to sell your house for $140,000. The advantage to a reserve auction is that it protects you better; the advantage to a no reserve auction is that it typically attracts more bidders and more excitement...and, hopefully, a higher price.
The seller is usually asked to provide some money up front for the auctioneer's marketing efforts--it can vary, but $3,000-$5,000 isn't unusual. The marketing is typically intense, with lots of signs, fliers, ads, even radio, for several weeks leading up to the auction. There are a number of different ways to conduct the auction, as well. If the marketing is effectively done, and expectations are reasonable, auctions are frequently, though not always, successful.
From what I've heard, usually the auction price of a property is a bit (5%-15%) under what it might have sold for using an agent.
As for who pays: Usually the purchaser pays a 10% buyer's premium. So let's take your $200,000 house. The winning bid is $170,000. The buyer would pay another $17,000 buyer's premium, bringing his total cost up to $187,000. Usually, the purchaser must have a deposit in certified funds at the time of the auction, and the remainder in 30 days.
Hope that helps.