Seller can back out as long as it's stipulated in contract. The bottom line is that if you don't understand your contract and your agent, then you should consult a real estate attorney to represent you. Are there two agents involved?
Firs try to go read over your contract with your agent to find out the conditions, contingencies, terms, amendments, and deadlines. And if it still doesn't make sense, then your real estate attorney could align your transaction.
By the way whatever you agreed with Seller and it's signed by both does not need to go through escrow in order to be valid.