Are you talking about 1) Buyer Agency Agreements or 2) Purchase Agreements?
(1) These are optional; you don't have to sign them. However, they're enforceable when you do sign them. Nevertheless, if you opt to not sign them, then be sure to use the agent who showed you a particular property to represent you on that property. Otherwise, you'll burn a bridge, ruin your reputation, and word will spread about you--if you do this multiple times--so that few other realtors will want to work with you in the future.
Many investors I know like to work with multiple realtors, because each realtor has a particular specialty and/or farm area. For example, some realtors are REO specialists, others might be pre-foreclosure specialists, and others might specialize in selling duplexes in a particular neighborhood. The point is you want to work with the right realtor for a specific deal. You don't want to use a pre-foreclosure specialist to help you close on REOs.
Some investors also invest on properties outside their local area. So, they also won't sign those agreements for obvious reasons.
(2) The way you exit these agreements will depend on the terms therein. Assuming you specified an inspection (or due diligence) period, and assuming that you haven't already waived all of your contingencies, then you could terminate your agreement (in writing) by exercising 1 or more of your contingency clauses. This way you'll also get your EMD back. However, if you've already waived all of your contingencies, then you still could terminate your contract in writing. You'll lose your EMD in this case, and--depending on the terms of your contract (including the addenda)--your action might possibly trigger an embedded 'failure to perform' clause. In either case, you should probably consult with an attorney to go over your options.
If you can remember the agents name and or office just call and ask for the agreement to be voided.
Now from here on out if you are not really willing to work with one agent just sign a buyers agreement for a short time frame. Like a week or so. Just be careful of signing more then one this can cause problems.