Executed contract you signed governs performance of buyer and seller should include what if's that buyer does not close. No matter what obtain all in writing via amendment to executed contract you released from sales offer contract.
You'll have to read your purchase contract in detail to see if it contains any contingencies that may let you get out of the deal. Since the build out has begun, my guess is that in a worst case scenario the seller could sue you to force you to move forward. At the very least they may be entitled to keep any earnest money deposits you've given them. Did you have a buyer's agent representing you? If so, then ask their advice and then you'll probably want to consult a real estate attorney
Best of luck