If you are unhappy with your agent fire him/her and move on. Don't verbally agree then try to get out of it.
When you have a moment, take a look at your listing contract. If it is an Exclusive Right To Sell contract, look at section A. It states that all parties understand that the broker should still receive commission if any event occurs, then you'll see a list of those events. Around lines 36-53. It will clearly define the answer to your question. Sub section 4 states that if "the property is sold, optioned, leased, or exchanged by the Seller or any other person within XX days after termination of the contract to any person procured in whole or in part by the efforts of the Broker, any cooperating broker, or Seller..."
In so many words, that's saying that if that agent was in fact the procuring cause in the transaction (providing the materials that got your property marketed and hopefully sold), they are still entitled to a commission. In my opinion, it would be in your best interest to pay the commission to the agent.