I would recommend doing a Google search for private lenders in your town and starting there. You will likely pay a premium as private lenders are willing to take more of a risk.
Call and ask questions.
May I ask you... What is it that is holding your credit down? Are they hardships that are passed you or do you currently have hardships and collections that are validly yours?
You would have to show that you are a good credit risk and that your hardships are behind you in just about any lending case.
I hope this helps you.
Lending since 2003 and ready to help...
With a 580 to 619 score, you will most likely need to show rent history with cancelled checks or 12 months bank statements with no more than 30 day late. The 3.5% down payment must be your money. It cannot be a gift or borrowed. You cannot have more than $1000 in open collections. You must have at least 3 lines of credit. At least 2 of those must be a year old and one can be 6 months old. You will probably regret doing the loan with this score because loan officers don't get paid what they used to get paid. Much of the mentality is: Why should I spend a lot of time working with someone with bad credit when I can spend less time with someone with okay credit and get paid the same?
So, if you are going to buy with this score, then I recommend that you buy a $250,000 house and make it worth their time. Just make sure you can afford it. In the mean time, work on your credit, your savings, your budget. Make sure your bank statements don't show you eating out all the time and that you aren't creating non sufficient funds on the account every month.