There are a few things you will need to get this done. You will need comps (comparable properties) available when the property is appraised. If there are no similar properties anywhere near you, it could be an issue. The appraiser will also have to make note that the highest and best use is as such (a residential single family home). You made mention that the town will let you build it again if need be, so you should be good there.
Let me know if you need further help. If the property is in NJ and you want me to take a further look into it let me know.