Technically, the terms of the contract weren't fulfilled but talk to your Realtor and tell them your concerns. I can tell you that statistically down payment has very little to do with performance. Programs like UDSA don't require any down payment but have lower foreclosure rates than other programs.
It's more important to assess the borrower's ability to repay the debt and their credit worthiness. Can they pay and will they pay are the real issues. Skin in the game feels like it should be important but not as much as you may think.
Speak to your Realtor then make your decision. If you're early in the deal then people will be disappointed but probably won't force you to sell your home. Talk to your Realtor.
You need to tell your Realtor your concerns as soon as possible. They represent you.