Dear Trulia Fan - Waltham Home Buyer -
If you don't want to walk away - get a third opinion from a Certified Mold Inspector. There are many different kinds of mold some toxic, some not. Some people have allergies, some people have small children that can be severly impacted by molds. As far as resale - there are definately some buyers who would not even look at a house with past mold issues based on their health status. Some who don't care, as long as it has been remediated.
Google Watham MA Certified Mold Inspectors - you will get a list of several companies.