I think the zoning person was only providing necessary information, statements of fact, to the appraiser, and "not" making any decision regarding your loan. To lend, or not to lend, is always up to a mortgage company.
The purpose of an appraisal when obtaining a mortgage is to protect the lender's interest. Of course it is beneficial to us as home buyers, but we are sitting in the back seat, along for the ride.
Grandfathered rights are never intended to be "eternal." Over the long haul, deterioration will always result in conformity.
Wishing you the best!