Have the buyers hired a lawyer who sent a letter to you or are the threatening to sue? If they have already hired a lawyer, they have escalated the process. If they have not, it would me prudent for you to meet with an attorney and have them write a letter that also lays out the financial repercussions for the buyer should they lose.
Do you know what the buyers want? Perhaps you can come to some reasonable accommodation that will make this problem go away, you can move on with your life.
I am an agent who sued a seller for fraud and I won. However, it took over a year, a lot of my time, and cost a total of $140,000, that the seller paid. Lawyers are obligated to go through mediation first, before it ever goes to trial. We went through mediation, the seller would not settle, so we went to trial. Seller ended up paying my court fees, his court fees, my lawyer, his lawyer, buying back the house. I was "made whole", seller lost beaucoup money.
Get a good attorney who will lay out scenarios and advise you. I hope it turns out well.