Your neighbor could be concerned at some point if you decided to claim that land.
When I lived in Brooklyn, the houses are pretty close together.. well, there was a house that had a shared driveway, well it was because the family that owned the property right up to the side of the neighbors house let the other people use the driveway and make a parking area behind their house.
Well, when they went to sell.. the people did not want to give up the driveway and parking behind their house.. They took the next door landowner to court and the Judge sided with them and made the property universal between the two properties, even thought the one person owned the land.. not good.
So, to avoid all the nonsense, move the fence.
Unwavering Commitment to Service
I would move it. I had a a situation where my buyer had a survey done as part of the purchasing process. The survey showed the neighbors fence was 5 feet into my buyers property with a shed there as well. The neighbor eventually moved the fence, but it got ugly for a while because the neighbor said he assumed" the property was his since the fence was there when he bought the property, We we almost had to take him to court to get an order for him to move the fence.
Do check to see if you have a survey of your property, then check with your local zoning official on where the fence is allowed. Some municipalities allow on the property line while other require a few inches inside the property line. If it's a wood fence, most municipalities also require the unfinished side faces inside, towards your house.
Unless it's a really really long piece of fencing, it shouldn't cost that much to move it, and will save you headaches if you (or your neighbor) eventually sell the house.
Here's the thing - you really don't want to get involved in an adverse possession lawsuit. The legal fees will be high, and the value of the decision low.
All the best,