That's not unusual. The decision on where to build a prison usually has nothing to do with the crime rate in the immediate area. And sometimes the nature of the neighborhood changes after a prison is built. In the Washington, D.C., area there was a prison in Lorton, Virginia. It generally housed inmates who came from Washington, D.C., about 20 miles away. Lorton itself started off as quite rural. Then evolved into a somewhat upscale area. Some folks in the area even had fun with it, putting signs outside their homes "Lorton Overlook." Nowadays--with the prison closed for a while, people new to the area don't associate "Lorton" with a prison at all. And there never was the assumption that because a prison was there, that the crime rate was high.
Hope that helps.