It has to do with an unbroken chain of events that lead to the sale of the property. Since you do not have confidence in this agent, and he has already given you wrong information, you can choose to "break the chain", by letting him know that his services are no longer needed, and finding and hiring a new agent.
If you feel generous, you might ask your new agent to agree to throw a "showing fee" (maybe $500) at the old agent, but other than that, since you have not signed an exclusive buyer's agreement, and he's given you wrong information... I would think that you are allowed to use the agent you want.
That being said, it's possible that this first agent will try to take it to mediation, and make a claim on part of the commission, but that's all "behind the scenes" stuff between the listing agency and the first agent's agency, and shouldn't impact your decision to change agents.