Contact the agent and his broker with your dissatisfaction. Do so calmly, and logically, and have any documentation of your complaints ready in writing.
The broker may want to attempt to keep you within the firm/office... think about that in advance, are you interested in staying within the office (you may have to see this current agent coming and going... will it be awkward??), or she may be totally apologetic, and willing to release you.
If she does release you, get the release in writing.