I've seen great discrepancies between high, low and middle scores. The banks decided, or I should say Fannie Mae & Freddie Mac who set the guidelines decided that that was the safe way to go. If you only have 2 scores, they choose the lower one.
Here's the good news, if you have a 650, you only need to work on one report to get your middle score up. Look at the accounts on your next best score; dispute any errors in writing to that agency. Pay down any balances to 1/3 or less of their high credit limits, DO NOT CLOSE any accounts, this will lower your score.
You may be able to improve one score in a few weeks on your own or in days with Rapid Rescore. Ask your lender about that program, it costs some money, but it's legitimate and it works.