It depends on how you want to play it out. There is no contract, so there is no obligation on your part. If you choose, you can not pay a dime and tell the agent to get paid from the buyer, but you run the risk of losing a buyer.
If you choose to pay, the question is how much? You'll get all kinds of different answers. My opinion: I'm not paying someone who came and knocked on my door with a buyer the same commission I'm paying someone when I will be expecting them to market my home until it sells. The end goal of the job is the same, however the means to that goal are very different.
Most FSBOs are open to paying a buyer's agent a commission for bringing a buyer. It makes sense because without them there might not have been a buyer. Without a buyer, you're not moving on to the next step in life. Your case could be unique because you may not need or want to sell.
Ultimately, who pays the commission is up to you. If you want to sell, consider paying it because the buyer might move on. If you don't need or want to sell, put it one the buyer and see how bad they want the home.