If the appraisal has not yet been performed and there has been no cost incurred, I would give you your money back. This is handled however each indivual loan agent or office wants to handle this.
Personally, I am not in this business to hold onto your appraisal fee! I am about good will and if you choose not to close for whatever reason, why not give you back your funds.
When you are ready to close, you will have a positive feel about that person and go back to them.
No refund? Bad feelings.