On the damage note, be careful how the repairs are completed. In my opinion, I'd get estimates and a credit rather than having the work done before closing. That way you get the warranty, and with the contractor of your choice. Using estimates and a credit/holdback will move the closing forward faster, too.
(The lister is just trying to make sure that "extra" damage does not happen to the property. For example, if you tried to move anything to survey the damage, or crawled onto the tree to get a better look and caused more damage than was originally caused by the tree falling, then the seller would have an issue with the damage that they need to be responsible for. If you exacerbate the damage, you could become responsible for all of it, so I'd be careful. Having an agent there with you will help everyone know who is responsible for what damage.)
The listing agent seems like a piece of work...The issue is not you having driven by...it's not as if you went physically into the property.
The issue is that a tree has fallen on the home and they need to address what if any damage has occured and how those damages are going to be repaired.
Getting upset or put off that you went by the home is a bit odd if you ask me.
You should refer to your P&S. There should be provisions in it for damage incidents as you've described. Your P&S should also have a provision for a final walk through of the property prior to your purchase. This is the legal right to enter the property.
Best of luck!