Some people are worried about living in a mixed socio-economic environment just like you. Others think it is an added value to get to know people that might do other types of work or have other needs in thier lives.
Each person has to decide for themselves what is right for them. You should go to the area at different times of the day and night. If you are worried then it isn't a good fit for you.
When you are buying a house it is really important that you feel OK or better about the area. Don't try to force yourself to do something that you aren't comfortable with.
I think that the issue isn't whether or not someone is on subsidized rent, it has more to do with how does the area look. If the proximity is in disrepair or just dirty then it will effect the property values. This is the case anywhere not just the area you mention. As Realtors we see it all the time. A house next door is vacant and in bad shape, it will hurt your sale.
As a community organizer it is the easiest bump in value to get done if you can get people to clean up an area or even just a single house.
The trend in urban design is to build mixed income housing. we see this in the Hunters Point build-out and in all the replacement of the housing department unitsunder the HOPE VI federal program and HOPE SF at North Point, Ceasar Chavez, and coming soon Potrero and Bayview/Hunters Point.
I do not think that is or will be a problem for many people. It will be for some, unfortunately. Go to http://www.sfgov.org to check out crime stats from the SFPD via the Mayor's website. I have never seen graffitti on those buildings and the tenants seem to take good care of them. Drive around the area at night and see how you feel. If you like it, go for it!
I hope that helps.