If the base of the tree is very close to the fence line, I would probably build up to it from either side and leave a little clearance. You can also choose to have the fence jog around it, but I wouldn't recommend that because that will result in your neighbor being able to legally claim that little area as his own some years down the road.
I am not sure if you are located in the city of Venice. They may have their own Code Enforcement Department.
Tanmmy Hayes, Realtor
RE/MAX Palm Realty