I am not sure if you are unhappy with your listing agent or buyer's agent so I addressed both situations. First , if you entered into an exclusive right of sale listing agreement with your broker to sell your home, then you cannot enter into another listing agreement with a new broker until the first listing agreement expires. You can withdraw the property from the market. Or if you want to stay with the brokerage company, but you don't like your agent, you can call the brokerage company and explain that you would like another agent in the company to handle the listing. Then it is up to the brokerage company to work the situation out with the two agents in their company, commissions, etc . Agreements are entered into with the brokerage company; the agent works for the broker unless the agent is the owner/broker as well. Read your listing agreement or have your attorney review it for you.
If you are unhappy with your buyer's agent, then just be honest and tell the agent you prefer to work with someone else that you feel is more dedicated or has more time to devote to you, unless you are already in a contract on a property that they are representing you on or they showed you a property that you are or were in negotiations with, and/or you decide to go back to that property and make another offer with your new broker. Things get complicated at that point because there may be a procuring cause issue between your new broker and your old broker over who is entitled to the selling agent's commission (the buyer's agent is the selling agent) if the deal goes through and you buy the property. To avoid any unpleasant sitautions, just be honest with all your brokers so they can decide between them how to work out any potential commission disputes over a property they both are claiming they were the procuring cause on. It's always best to end the relationship in a friendly business manner because you never know when you may end up in a negotiation on a property with that agent again on the other side of a deal. They may represent a listing that you want to buy down the road. Also, if you signed a buyer's agent loyalty agreement with your old agent stating that you agreed to have your agent exclusively represent you in all negotiations to purchase property or pay a commission anyway for property that that the agent showed you during the contract period that you bought, you could be liable for that commission. Again, read your buyer's agent loyalty agreement or have your attorney review it.