There may be other issues with getting a loan but I think you need to talk with several lenders and look at all your financing options. Don't listen to just one person - they may not know all options. The selling bank may not offer a construction loan, but another lender may. Talk to more lenders.
Most people complain that the interest is too high or it costs to much to use a hard lender. When I hear them I always say... would you rather pay 16% interest or 50% to a partner? The numbers answer that problem.