JB, State laws vary and I'm not an attorney, but in most cases purchase and sale contracts on residential properties have multiple escape clauses from them during the early phases. Inspection addenda, Title contingency, etc.
A buyers agency agreement may not have this built in but I can't imagine an agent trying to hold someone against their will unless they worked and earned a commission.
For example, you meet an agent on Monday and sign a buyers agency agreement. They do some preliminary work for you but you haven't seen a home with them yet nor have you signed a purchase and sale agreement. You decide not to use that agent and give them notice. Chances are they try to repair the relationship but then move on.
In another arrangement you sign the buyerâ€™s agency, they show you a number of homes, you like one and write an offer. An unscrupulous listing agent says, drop your agent and I'll get you a better deal on the house. This would be unethical and a violation of the contract.
As in all cases, know what you sign before you sign it and if you are not sure, have an attorney review the document with you first.