No, when you lease or rent someone's property it become temporily your. There are State laws that protects renters. Unless it is stated in your rental contract. Check out your State laws for renters rights on lease property.
Your question is vague but I will attempt to answer any way. Is he calling you ahead to let you know he is coming? "Recreational day"What exactly is he doing in the yard? Landlords do have the right to come at different intervals throughout the year to inspect the property and this is usually stipulated in the lease, but popping up randomly, without 24 hour notice (minimum), is not appropriate. Tenants have the right to "quiet enjoyment" of the premises. My suggestion would be to talk to your landlord and see if the two of you can come to an agreement on how to handle future visits so that you feel more comfortable.