I suppose the lender or the appraiser could bill you for the cost of the appraisal and maybe the credit report, but that's probably about it.
If you werent comfortable with the terms of the loan or the lenders explanation of them then you were right to walk away. The worst case is that you keep the loan you have and you still have a roof over your head. The only bad part is that 3 weeks ago was about the low point as far as rates, and they have gone up since then so good luck finding rates THAT low.
That's not to say rates aren't still low and it won't make sense to refinance. Talk to a local,reputable lender who can help you work out your options.