Your contingency removal does not relieve the seller of his/her responsibility to make agreed upon repairs. Essentially, the seller does not have to agree to make repairs and would not want to make even agreed upon repairs without an assurance that the transaction will close. As long as you have not removed your contingencies, you are still free to cancel the agreement and get your deposit back. If you are concerned about the seller not making the promised repairs, this should be addressed in an addendum to the contract. For instance, you could agree that money be held in escrow until the repairs are completed, which would mean that the seller does not get all his/her money until the repairs are completed. This will only work if the repairs don't have to be completed as a condition of the loan that you may be getting to finance the purchase. If the repairs are a condition of the loan, then you could remove your loan contingency with the exception of repair related loan conditions. The repair addendum should state a deadline by which the repairs need to be completed and your final walk-through should be after that deadline. The purpose of the final walk-through is to check whether agreed repairs have been completed and whether the property is in substantially the same condition it was when you made the offer (aside from the items that were repaired).
Just out of curiosity. Do you have your own buyer's agent? From the way your question reads, it sounds like the listing agent is also your agent since you said that the seller's agent asked you to sign the contingency removal form. If you had your own agent, the listing agent would not be speaking to you directly about this. If you do have your own agent, you should discuss your concerns with your agent. If you do not have your own agent, you may want to consult with an attorney to make sure your rights are adequately protected under the purchase agreement.