It doesn't sound so smart at this point;
"He has been increasingly difficult..." is a really poor reason for withdrawing from such an important thing.
The only question you need to answer is; Do you want the house?
Sometimes financing is held up and if we was counting on the funds from settlement within a certain time frame as outlined in the contract he may have reason to be frustrated. I say you hang in there!
What does your contract say? Each transaction is unique and while we use some standardized forms in many states, no one can give you an absolute answer without reviewing the paperwork.
Next, why would you want to do that? Starting over with another home and the seller looking for another buyer may take longer than waiting out this process. Yes, it's frustrating to be in a state of uncertainty, but before you both jump ship, gain as much information as possible so you land where you want to.
Chances are when it's a mutual decision, there are no penalties because there are no aggrieved parties (except possibly the agents, review any agreements you both have with them too). Just don't forget you've likely already spent money for an inspection, appraisal and they have missed a significant amount of marketing time. Do they have a back-up buyer? That may explain their desires.