There seems to be some information missing from your question though. You should definitely talk to a lawyer and if the agent did bring you a ready willing and able buyer, one could argue that they did their job.
Tricky situation... but without all the details no one can really give you any advice but there is also a provision about mediation that you could look into in your contract.
But to answer your question, 'can they sue me?' as BJ said, anyone can sue anyone..