I think it's wise to borrow as much money at 4% or under as possible, especially if you have other opportunities.
First off - you should have an accountant. If you're a homeowner, you should have an accountant. Talk to them about ways to plan for the future.
Here's one scenario. Inflation is a straight 2% for the next 15 years. That means that the real value of money will decline 35% - your $1000/mo mortgage payment will feel like $650 in 15 years.
If you don't prepay your mortgage, you'll owe about 2/3rds of the original amount in 15 years. However - at that rate of inflation, it will "feel" as if you're more than halfway paid off.
What is the "real" cost of your money? In the 28% tax bracket, if you're paying 4% interest on a loan, the 28% tax break knocks it down to 3%, and with 2% inflation, it's essentially 1% money.
So now the question is: can I do better than that, in real terms, with my money? Maybe an inflation-protected annuity. I don't know what you -should- do with your money, but I would strongly consider investing it outside of your house rather than inside of your house. Consider. Not "do," but "think about."
All the best,