They normally do not refund that charge, BUT you should have signed a form titled "Notice of Right to Reveive Appraisal." You can use that to obtain a copy, and have it switched to a lender of your choice for a small fee.
Not all Mortgage Brokers are equal!
I am a mortgage Broker. If I can be of further assistance, please let me know.