To qualify for a traditional loan, you are going to need a minimum credit score in the 620 - 640 range though most require 680. USDA loans with a credit score of less than 580 are denied if you have been late on any payments. (Reference: http://www.rurdev.usda.gov/SupportDocuments/CA-SFH-GRHUnderw
There are other loan programs out there that may accept less. One of these is FHA. FHA's website states: "Minimum credit score requirements for FHA home loans depend on which FHA loan product the applicant needs. Generally speaking, to get maximum financing on typical new home purchases, applicants should have a credit score of 580 or better. Those with credit scores between 500 and 579 are, according the the FHA guidelines, 'limited to 90 percent LTV'." (Reference: http://www.fha.com/fha_article.cfm?id=200.
You could look into rent to own, land contracts or sub2 agreements.