Some people are worried about living in a mixed socio-economic environment just like you. Others think it is an added value to get to know people that might do other types of work or have other needs in thier lives.
Each person has to decide for themselves what is right for them. You should go to the area at different times of the day and night. If you are worried then it isn't a good fit for you.
When you are buying a house it is really important that you feel OK or better about the area. Don't try to force yourself to do something that you aren't comfortable with.
I think that the issue isn't whether or not someone is on subsidized rent, it has more to do with how does the area look. If the proximity is in disrepair or just dirty then it will effect the property values. This is the case anywhere not just the area you mention. As Realtors we see it all the time. A house next door is vacant and in bad shape, it will hurt your sale.
As a community organizer it is the easiest bump in value to get done if you can get people to clean up an area or even just a single house.