In the instance of a seller finding his/her own buyer, you should have an Exclusive Agency contract in place, otherwise the listing contract for PA states that a commission is due to the broker regardless of who brings the buyer to the transaction. (And many times the buyer finds out about the home or becomes interested in the home due to the broker's marketing efforts and then goes directly to the seller...but it was the efforts of the broker that brought that buyer to the seller!)
In the instance of the agent not performing, then you should go to the broker and have a chat. Perhaps a different agent can be assigned to your listing.
It is rare that a broker will take the time and money to pursue legal action against a seller over a commission but it most definitely could happen. And no real estate agent worth the paper their license was issued on will take a listing that has a current and valid contract with another broker.
Good luck! You may want to involve an attorney if talking to the broker doesn't help.
It takes time and money to sue someone, so unless the broker has a very good case, there is a good chance that they will just not pursue this.
You will never be 100% sure that you will not get sued.
I can tell you that here in California, if you are not under a contract with a Buyer most Brokerages will let you out of the listing contract providing for certain contingencies. And that is spelled out in the Listing Agreement and Cancellation of Listing Agreement.
Discuss everything with your Broker. They will want to do what they can to make you happy. If that can't happen, they can not make you sell your home, they can, however, hold you to the contract and not allow you to list with someone else (this is rare in our area).
Good luck. All the best to you.