It is important to use a Real Estate attorney who specializes only in Real Estate to help protect your interests in this tranactions
Your attorney should discuss how much is needed on contract. It is traditionally 10% if you are going for a conventional mortgage.
In NY the contract money is held in an escrow account by the sellers' attorney. Your contract specifically states the conditions on which the contract money is refundable. If you haven't hired a real estate attorney, do it immediately.
Understand that it is also called "Earnest Money" or "Good Faith Money" and the more you put down, the stronger it looks to the Seller.
If you are putting 20% down, you will need to put even more than that in the Escrow before you're through.
If you think you will back out of the Deal, don't start it.
If you do a good job with the Contingencies you should be Okay.
If you can, put down as long a Contingency for the Loan as you can, anything can happen.
The Appraisal and Inspection Contingencies will protect you.
Let your Buyer's Agent guide you on how much.
Good luck and may God bless
Congratulations on purchasing and best of luck,