Hello each and every contract is written differently so it is best to reread it yourself a few times, looking for a performance clause. I would then suggest you contact the Attorney General's Office in Boston who is the first line of consumer defense in this kind of matter. You may also want to double check the status of the contractor's license and insurance and if they are not intact then it may make it easier to remove yourself from the contract. Most likely, unless the contractor has ordered supplies or invested in custom made items for this project, they too would want to remove themself from a work situation where even before the project has begun, levels of expectations have not been met. Your comfort zone has been violated regarding the project.
If all else fails check with your local attorney to review the documents but I would say it is a matter of you responding in a timely fashion and also documenting all that has transpired. Good luck and if you do wish to consider other home remodelers in your area I have a referral base which is based upon my Clients being well satisfied with their expectations met or in most cases well exceeded! Just give me a call.