You didn't provide many details about your ARM. You might be surprised how little more you spend on interest after converting to a fixed loan. You have to decide your own tolerance to the risk carrying an ARM.
One advantage of low interest rate loans, if they are paid at a fully amortized rate, is that you will be building equity much quicker early on than with a higher interest rate loan even though your payments are lower. Please take a look at:
Keller Williams Realty
There should be no other money you will lose.
If your new loan was a no points no cost loan, you should take it and ride with that one. You can always refinance if rates go lower...how could they?