It is difficult to determine based on the information you have given. That said, I assume that the buyer / buyers agent / contractor got permission to enter the property from either the seller / sellers agent.
If not then they could be considered to be trespassing.
Would it be grounds to not perform under your contract, I would ask my legal counsel.
If you decided not to proceed with the contract based on this, you could be tied up in court for a long time, with large legal bills.