Most lenders require a 640 middle score for USDA but some of us can go down to 620. The best approach at this point is to work with a mortgage professional that can give you some direction on how to bring your scores up. It maybe something as simple as paying down your credit card balances and doing a rapid re-score.
Are you both looking to qualify using each of your incomes to pay back the loan?
Where did you get your 'scores' from?
Di you have middle scores?