Yes, you can terminate your representation agreement, but the question of "why" remains, and the question of whether the broker has earned any payments under the agreement remains.
If the broker has not lived up to his side of the agreement, then speak to him and explain why you want to terminate. Normally, Realtors do not want to hold clients against their will.
However, it is possible that because he fulfilled his side of the agreement that you owe him payment of commission earned. Terminating the agreement does not relieve you of an amount already earned, and you should be prepared to make good on your side of the agreement by paying that amount. If there is a contract still in execution, you will still be obligated to pay him, even though you will not get any benefit of representation once you terminate the agreement.
Most representation agreements also contain a 'protection clause' which prevents clients from reaching an agreement on a property and then discarding the broker so that they don't have to pay him. If, for example, the representation were to end on January 31, and you went back to a seller that you previously negotiated with and signed a new contract on February 1, the broker would probably still be owed a commission under the protection clause.
Your first order of business is to discuss your issue(s) with the broker and try to resolve them. If you can't resolve them, then tell him in writing you want to terminate the agreement, keeping in mind that if your objective is to avoid paying a commission on a property he introduced you to that you're not acting fairly and legal repercussions may result. After you discuss the matter with the broker, you can still hire an attorney to help you understand your rights and obligations and to help you resolve the matter.