You have to have paid it down to 80% of the appraised value from the time you took out the loan or get an appraisal showing that you have 20% equity.
If you can't do one or the other then the m.I. will be required.
The occasional late payment shouldn't effect things. It's based on value. Ask your lender about the actual process to get it removed.