You're allowed to borrow money from an IRA for certain kinds of transactions, but not for others. I don't think this is one of them, because I believe it's not an arm's length transaction.
Here's another option that you might not have considered. Use your IRA funds to acquire 1 or more properties (which don't necessarily have to be in your area) to flip/rent to increase the amount and/or yield of your IRA. You could borrow against those funds for your business venture (of course you'd have to work with knowledgeable pros [including a CPA, attorney, etc] to help you set this up properly).