It sounds like you are buying a home with a Realtor. Your question should be posed to him or her.
In general once an offer is accepted, your check is deposited with an escrow company, who acts as a third party intermediary. During the "contingency period" you are supposed to obtain your financing, make sure that the property appraises for the loan value, have the home inspected, etc.
During that time you as the buyer can pretty much cancel with no penalty at any time. Once the contingencies have been removed, then you are pretty much committed unless something very unexpected happens. Now it is common for inspections to produce repairs that need to be completed for the buyer to agree to proceed. It is also a point when everyone in the transaction finds out how much the seller wants to sell, and how badly the buyer wants to buy.
If the seller will not complete repairs that you think need to be completed, then you "should" be able to cancel with no penalty (just my opinion, this is not legal advice. For legal advice see an attorney).
If your Realtor is not helpful ask to speak with their broker.
Escrow may be charghing you their fee because of the work they have put in to produce escrow instruction, order title reports and other things. This is an inexpensive price to pay to get out of a transaction where the seller can't address items to be fixed.
I do find it odd that the escrow company is charging you for their fees. They certainly may do so, but the escrow companies that I deal with do not charge buyer if he or she decides to back out during the inspection period.
Hope this helps,