Could it be that there was termite damage before you hired the termite company to treat for them? Did the buyers ask for an inspection by a competitor after a home inspector found damage in the joists?
The fact that an inspector could knock holes in your joists indicates at one time there were active termites. Perhaps that was the reason you contracted with your termite company in the first place. The questions these buyers most likely wanted answers to are 1.) How extensive is the damage to the joists-has it compromised the structural integrity of the house, or do any joists need replacing, etc?, and 2.) is there any evidence of active nests now?
As you are finding out, buyers want to know exactly what they are buying before they plunk down their cash and take out a mortgage. Can you blame them?
If I read your question correctly, "knocking holes in my joists" suggests they were indeed able to knock holes in your joist. I am neither an inspector or builder, but I do not believe a joist should be easily perforated.
If I have read your question correctly, the inspector simply revealed an issue the needs to be corrected.
If however, they aggressively attempted to create a hole and failed leaving behind evidence and damage, now that's a different story.