This I admit is an unusual question. However, my question back to you would be. Have you invited them in? Either by requesting a reduction in taxes or by requesting a building permit. They cannot legally enter your home without probable cause unless they feel that you are in danger in some way. Which is the premiss under which some communities have gained access to neighborhood homes. In most cases, they will not enter your home without you being present or letting them in.
Hope this helps!