Some lenders will allow you to do an FHA loan down to a 620.
In order to improve your credit, the best thing you can do is to get a small credit. Get something with a limit around $500. Then just put gas on it every month and pay it down to a dollar. Using the card is the quickest way to build your credit. With just your student loans, the most difficult thing for you to overcome is the lack of credit history.
If you are a veteran, some lenders can go down to a 580 credit score.
I will be glad to answer anymore questions you have.