As others have said, a 580 credit score is the rock-bottom that's required to get a mortgage, but there's still a way to get through that if your score is below that. There are several programs in and around Philadelphia that are designed to help people in your situation to get a mortgage/find help with down payment assistance, and many of them simply require that you attend a series of (free) credit/mortgage counseling classes.
If you'd like more information, feel free to contact me! Take care, and good luck!
Keller Williams - Center City Realty