I am a native of Fairfield County, CT but I know many people who are from Bergen County, NJ and Westchester, Putnam and even Dutchess County, NY. Most people decide between CT/Westchester and NJ/Long Island by which train station they work closest to. People in Wall Street come into Penn Station and take New Jersey Transit or LIRR to get there. People who are commuting to Mid-Town come into Grand Central which is fed by Metro-North.
When my family decided to move (we lived in Darien and Trumbull) we moved way down the shoreline to Madison which still had a large population of NYC Commuters. As simple as it sounds, pick your train station and follow the lines out. Towns like Westport, Fairfield, Southport, CT are beautiful but you'll pay waterfront prices. Trumbull, Wilton, etc... are one layer in from the water and more reasonable. Westchester County is also very expensive, so maybe going to Putnam or Dutchess will make more value sense.
Of course, most communities in LI (Cold Spring Harbor, Garden City, etc..) are just as beautiful and accessible to the city. Same with Jersey and towns like Upper Saddle River and the surrounding towns.
You have so many options, and so many transportation choices. Most of these towns are all populated by NYC commuters and have very strong school systems, good safety records and all of the other desirable features buyers look for. At the end, take the location that will allow you the simplest access to and from work so you can spend as much time with your family as possible.