Always a difficult question for other agents to answer. First, do you feel like you've given this agent really honest feedback about what they're doing or not doing that doesn't work for you, and the chance to redeem him/herself? If the answer is no, try that first.
If yes, and you feel you've done everything you could/should, then tell them honestly that it's just not working out and that you want to terminate your agency agreement. There is even a form for it, Standard form #720, "Termination of Agency Agreement & Release". If the agent is not agreeable or helpful, then call their office and ask to speak to their broker-in-charge.
If they have spent a lot of time with you, and you just haven't been able to connect personally or professionally for some reason, it would be a kindness for you to ask the agent you end up working with if they would pay some sort of referral fee to them. We've all had clients we just didn't click with, and this agent will probably be relieved at some level that you've been honest, and have moved on.