I'm assuming when you say "my contract" that you've agreed to sell your house to a buyer . . . that the contract is the sales agreement. If so . . .
One thing to take into consideration is that you may owe your listing agent his/her commission. Check the listing agreement.
Technically, you could be sued by the buyers. I'm not a lawyer, so this isn't legal advice. It's unlikely you'd be sued (for breach of contract or for specific performance), but it could happen.
You should check with your agent to see if you have any "outs." For example, if the buyers didn't apply for financing within the required time period, you might (depending on your local contract) be able to terminate the contract. There are certain other things they're required to do; if they didn't do them, you may have an exit strategy.
On the other hand, if by "my contract" you mean your listing agreement with your agent, then talk to the agent. Tell the agent you wish to terminate the agreement. Although it's a bilateral agreement, most agents will agree to terminate an agreement. If you run into any problems there, speak to the agent's broker.
Hope that helps.