Marc is correct in that the bank has the final decision on your offer to purchase the home; not the sellers.
Check your contract with the sellers regarding the deposit monies and at what point you are liable for losing it.
Who were the deposit checks made out to?
Do you have the capability of just cancelling them?
The banker says that you can walk so why is your agent saying that your deposit is at risk?
I suggest you discuss it again with your agent and review your contract with the seller.
Prudential Connecticut Realty
Good luck to you!