You need to look for a new lender who will lend in this complex.
I agree with Bob below. I would talk to a good local real estate attorney. This is a unique problem and the best solution seems to be to purchase the other unit with 20% down as an investment property. A good ARM could work well. You will need a team of a good Realtor, an expert lender, and a good real estate attorney. Good success with your venture.
Your Castle Realestate