I haven't found a lender that will do a rural, USDA loan under a 620 middle score. However, I do know of 2 seperate lenders that will do FHA loans down to a 600 FICO (96.5% LTV loan) & 580 FICO (95% LTV loan)respectively. More scrutiny will be placed upon your credit in these instances though since they are effectively 'portfolio' FHA lenders. Your prior lender is probably restricted to going down to only a 640 FICO as most Lenders will only go that low for an FHA (but not all).
I'm happy to answer any questions you might have and why you didn't recieve an approval with your previous lender. I hope you find what your looking for. Good luck.
Maker Capital Group, LLC