Who is the Realtor representing your interests? Have you purchased directly through the seller's Realtor?
Your Realtor can be there to represent you and relay the findings to you. If you do not have anyone to represent you and the schedule does not interfere with mine, I will go there as your representative...as a service.
If you hire a dependable inspector, they can do the job and report to you; it is not necessary for you to be there. If you do not have anyone you can depend upon, I will to provide referrals for you.
The listing agent understandably wants the inspection as quickly as possible; if for any reason you are not willing to go forward with the deal, they need to accommodate the next in line.