Im a young female and I just bought a condo in this area. Every day I walk 20 minutes to and from the Grove St path alone and I have never felt in danger. My block is nice while the surrounding area looks a little more shady. I think the area is going to get better, and in reality, crime can happen anywhere, there are incidences in downtown all the time and that is the "nice" area.
My advice would be to visit the particular block your interested in at different times to get a feel for it. If everyone took the attitude of fear, places like Hoboken and Downtown JC would still be a wasteland instead of the exclusive places they are now.