Neel, Charles is right, it totally depends on what you signed and who was the procurring cause.
Also, i would be cautious about using the phrase "not happy with" it's very vague. Find concrete examples of where they shirked their responsibilities, and weren't doing their job.
If you're really not happy with your agent, but don't want to get sued, ask your agents broker to assign you a new agent.
None of the Realtors have given you the right answer.
If you haven't signed a buyer agent agreement, you're OK. The brokers however, are not. The broker who showed you the house can go after the commission of broker that wrote up the offer. The broker who showed you the house can say "I showed the buyer the house, I deserve part or all of the commission". A broker is not supposed to write up an offer with you if he knows you've already seen it with another broker.
If you did sign a buyer agency agreement with broker A and wrote up an offer with broker B, you might owe a commission to broker A. And by the way, broker B knows this.
You might want to bring everything you've signed to a real estate attorney before you go any further.
A similar situation happened to me. I walked into an open house and did not tell the broker on duty that I was working with another broker. This was years ago before there was such a thing as buyer & sellers brokers. My broker told me that if I wrote up an offer with him and the broker at the open house recognized my name, he might not get paid. My broker told me in the future, to write his name into the open house register next to mine.
What about common courtesy? Buyer agency agreements and procuring cause aside, if an agent found you the house and/or took the time to get into their car, compile the information and meet you at the house without getting a signed buyer's agreement from you, it would be inconsiderate for you to circumvent them and go directly to the seller's agent.
My guess is that you think you can get the house for less money if you cut out your agent. Normally, if a seller negotiates their fee with their agent, the agent offers a portion (maybe half) of that fee to buyers agents to bring buyers. If the buyer goes directly to the listing agent, not only do they not have legal representation, they lose negotiating power because the listing agent's job (legally) is to represent the seller and bring the seller the best offer. You can't change the previous advice the agent has given the seller AND the listing agent can keep 100% of the negotiated commission. You can actually lose in this situation.
They also have an option of going after sellers agent first which usually happens, but if that falls through you are the next in line.
Like lots of things in life...it depends. Were you in an exclusive buyer's agency relationship with the buyer's agent? My short answer would be until you are in an exclusive agency relationship (meaning you've signed a buyer's representation agreement)--you are free to work with a realtor of your choice. My advice would be to interview two or three buyer's agents and then hire one of them to work on your behalf. It takes the head ache out of the process. You may still "find" a property on your own since most buyers are searching online--but the tricky time of the transaction is in the negotiation of the contract and the contract to close time period. Let a good agents expertise go to work for you!
If you really like the house and want to persue it, find a buyers agent that you can work with. Explain the situation and ask them to contact the original agent to work out a referral fee (the commission is coming from the buyers proceeds so it really doesn't effect you.) The original agent will sign a referral contract with your new agent and you should be covered to proceed with the agent of your choosing.
Most agents will accept the smaller fee to avoid all of the problems that arise from arbitration/litigation.
You may think you know the market and how to negotiate price and contingencies, but believe me, you will make a better deal by having the input of a good buyer's agent. Best of luck!
If you are truly unhappy you must break the chain of events with that buyers agent immediately. I do agree with charles, i wouldn't dump the buyers agent and work with the sellers agent just to get a better deal because that sellers agent will then be getting paid twice the amount and will be obligated to the seller during negotiations.
My sincere suggestion, do not dump a buyers agent to work with a sellers agent. Dump buyers agent for buyers agent. It is well worth your best interests being protected by someone who is not looking to get you to pay more money (the sellers' agents job) and instead someone who is representing your best interests (sellers agent is legally obligated to the seller, not buyer).
It all depends. Did you sign any agency paperwork with the buyers agent? Who provided the first info on the particular house, you or the agent? Basically, if you did not sign any buyer agency paperwork with the agent, you are free to move on to any agent you want to deal with. However, if the buyers agent is the one that first showed you the house, if he is the one that first mentioned this particular house, then if you buy it he is still entitled to a commission since he could argue that he was the procurring cause.
Best way to proceed would be to let the buyers agent know your concerns. Trying to eliminate him just to get a better deal from the seller will not work. (the sellers agent will likely try to keep the whole commission) If you absolutly can not work with this particular agent, ask for a referral to another agent from his office.