The borders for Sag Harbor are the bay, to the north (or the North Haven bridge), where Hampton Street becomes Route 114 (just past Lincoln Street) to the east, where Madison Street and Middle Line Highway intersect to the south and just past the light at the intersection of Main Street and Jermain Avenue to the west and southwest. These points will give you a general idea of the radius of the Village of Sag Harbor.
What you are coming across online is homes that are not in the Village of Sag Harbor (in many ways an independent entity with its own police station, mayor, taxes, ect) but rather in the 11963 zip code of Sag Harbor and the Sag Harbor School Disctrict, which has a much wider radius.
If you type in 11963 on this website, it will give you a better idea of Sag Harbor as a whole:
Similar situations exist in Southampton and East Hampton, where there are the Villages proper, but also the outlying areas (for example the Northwest Woods and The Springs in East Hampton) that still have East Hampton or Southampton addresses but lie outside of the "Village".
I have lived in Sag Harbor for almost 6 years (first in the Village and now just outside of the village) and have a great passion for the area. If you have any further questions, please do not hesitate to contact me at 917-691-0852 or via email at email@example.com.
Thank you for your inquiry and your interest in our area.