Do you have a mortgage on the house in Norfolk? If so, then you can't simply swap. You would need to sell the home.
If you own the home free and clear, then I suppose you could swap it. But that would be predicated on finding someone else who also owned their home free and clear and was interested in moving to Norfolk. The homes would also have to be similar in value.
That sounds a bit to convoluted, and even if you could find this situation would you really want to be pigeon-holed as far as the new house? Wouldn't you be better off just selling your house and purchasing any available home in Patterson?
Think about it. I hope this helps.