Good luck to you!
Best of Luck!
If you are purchasing in our area, our in house lenders have a program to review your credit score. It's possible they may be able assist you in bringing that number up relatively easy.
Don't give up yet.
I assume you are saying this because as pointed out already, your down payment is fine to avoid PMI, but Fannie Mae has a minimum credit score so you will not be eligible for conventional financing with that score. This means that FHA would be your only option and with FHA you will pay mortgage insurance regardless of your down payment.
Either consider paying the MIP (as it's called with FHA) or work on a plan to raise your scores. If you need assistance with either, feel free to contact me.
I agree with my colleagues, your down payment is fine it's your credit that may hold you back. Wells Fargo recently lowered their scoring models and may worth a shot for a conventional loan.
Your other options would be to do some credit repair, there are options which can go fairly quickly through legitimate sources or find a hard money lender.
Hard money loans are more expensive, higher rates and generally for short terms, a few months to a couple years. Only consider this source if you fully examine all the costs and can be certain you can get conventional financing soon and are getting an exceptional deal on the house.
40K down on a 133K house is more than 20 percent down so you should have no problem avoiding the PMI problem. Contact a local lender for lending options with a 580 credit score. Trulia has many loan professionals that I'm sure are willing to help!