Dear John, The property needs to build equity before you can refinance, so it is not a set time, it depends in the economy. Most lenders will lend you 80% of the appraised value. Also, what is the reason you need to refinance, if your interest rate is at 4% most likely the interest rates are going up, not down.
I recommend refinancing only if you are getting an interest rate 1.5 less than what you have, to make it worthwhile your expense.
I am a realtor with 26 years in the business, I hope my information helps you.