Honestly, if my client (the buyer) is not interested in the property - we move on.
There's no need to contact the listing agent if we're not interested.
As a Listing Agent, and am aware of what agent came through my listing with buyer's, I gather the business cards and contact the buyer's agent to get feedback. Usually my seller will let me know if there are cards left on the counter.
I feel it's my job to do this. The buyer's agents are usually very honest & give me feedback on what their thoughts are. Was the property listed to high? They were in other homes with more upgrades & amentities. Location? etc., etc. I can turn some of their feedback into a more positive experience for the next buyer.
If I receive positive info., I pass it along to my seller. If it's "constructive" negatives, like the house needed to be cleaned up, etc. I will pass that along too. Nothing ever offensive.
Best of luck with the sale of your home!