It depends on your situation and how long you are planning to stay in the home. Refinancing is not "free" - regardless of the advertisements you hear claiming "no closing costs" or "absolutely no fees". The fact is that there is absolutely no mortgage broker or lender in the world that will simply lower your interest rate, without tacking on some type of origination fee, olling that amount into your loan - raising your principal, or charging you a higher interest. They wouldn't make any money if they did.
If it's going to cost you $4000 to refinance into a lower rate, and knock $100 per month off your payment - you aren't going to "actually" see any of those savings and recoup the money you spent on that fee for over 3 years. If you are going to be staying in that house for the long-term, i.e. 10-15 years, then you'd be saving a fortune in interest and it makes sense. If you are uncertain about your future, and may sell the house and move in 3 years - then you would be actually losing money by refinancing in the long run and it makes no sense.
You'll want to shop around for the lower rates, and lowest associated "fees" before making a decision. Once you have an idea of the true cost, you'll want to crunch the numbers and make sure that the amount of reduction in interest, justifies any associated fees, in correlation with how long you plan to remain in your property.
Hope that helps!