The home inspector's job is to identify that a problem may exist. You need to investigate further. Were there just a few and where are they living? Certain times of the year, they go out and explore for new diggs. Seeing ants alone does not always indicate a serious problem. Investigate further. If they are indeed infesting your dwelling, A contractor or structural engineer can tell you how to fix it and what it will cost.
If you have carpenter ants, that means you have a moisture problem. Carpenter ants will move on if the water problem is fixed. You need to hire a contractor to fix the water problem and also fix any damage caused by the ants. The ant damage may be minor or great depending how long they have been active. Carpenter ants eat rotten wood, wood rots because of moisture and mold.
Move on if the seller will not give you further access to the problem or will not certify the problems have been properly fixed by a qualified engineer or contractor.
"-- Would you buy a house like this (assume everything else is fine). "
This would not, in my opinion, be the single reason to buy or not to buy a home. You now know you have a problem, and you need to find out what the detrimental value is. Work with the owners to identify and resolve the problem.
-- If I buy this house, would this affect its resale value/prospect in the future?
It has already affected the resale value. If the water and ant problem is not dealt with, it will likely get worse and continue to negatively effect the property.
No cause for alarm just request that the seller remedy the problem.
Living in Denville as in any cmmunity in NJ, you will find "something". I cannot tell you the % of carpenter ants in Denville, but termites are prevelant and curable. It will not affect resale in the least.
If that is the only thing, be thankful and start your life in this wonderful town. I hope I run into you at Denville Dairy.
I had a carpenter ant problem in a rental house of mine. And it was no big deal. I had the house treated, and the area inspected for structural problems (there weren't any). A pest control person can also tell you what to do, in addition to treating the house, to minimize the likelihood of the problem recurring. For instance, trees that overhang the house provide a way for carpenter ants to get to the house.
To answer your specific questions:
What should you do? As noted above, get a pest control expert in to examine the situation.
Should this be a cause to move on? No. In fact, even if there's structural damage, you determine how much it is, then have the seller pay for it.
Would I buy a house like this, assuming everything else is fine? Yes.
How common is this problem? I don't know. But it's not uncommon.
If you buy the house, would it affect the resale value? It shouldn't. At worst, there would be structural damage which would be fully repaired. But, again, that's very unlikely. Most likely, there's no structural damage and, frankly, little visible damage.
So, get a competent pest control person out to take a look at it. But, really, don't worry about it.
Hope that helps.