Yes you can terminate the Agreement. First of all, the Broker in Charge / Office Manager is responsible for their agent's actions. They also must sign the Agreement since the brokerage owns the listing - not the individual agent. Also your listing is with the brokerage - not the agent.
You need to speak with the Office Manager / Broker in Charge and ask to be released from the brokerage contract. You can stay with the brokerage and be reassigned to a different agent. Most managers will prefer that you be reassigned to a different agent so you don't leave the brokerage altogether.
Bottom line - do not stick with your current agent. Selling a home is stressful enough. Your agent's behavior doesn't need to add more stress.