If you are not happy with the service your agent is providing, and have signed an agreement for representation. ( buyer representation), you may indeed terminate the contract.. Did you Read everything you signed before you agreed to work with that particular agent? Did that document spell out the duties they had to you? Did they interview you about your housing needs, help you secure a lender, explain representation? Finally- did they show you homes in your price range in the areas you requested to see? If they did not do any of the above listed items.. you should contact their broker and ask how to terminate the relationship , then do so in writing immediately.
If the broker or agents does tells you that you cannot terminate- contact an attorney.
It's always best to start with a clear understanding and expectation about what your agent will do for you. Once that 'job' description is fully described, you should have no question ( nor should they) if they have not provided proper service. FYI: You can always add language to a buyer agency agreement that states:' This contract may be terminated with written notice ".
In my 15 years as an agent, I have found that good agents will openly share this with you and weak agents will try and use the buyer contract to keep you.
All that being said.. if you have written an offer in place ... or are trying to go behind your agent's back in order to 'get a deal directly from a seller... .. expect that you could be responsible for a commission.
If all buyers, sellers and agents had honest and ethical dialogues everyone would benefit.