Yes you most certainly can get a mortgage with your credit score but I would advise you strongly to keep working on your credit because credit issues have a severe impact.
So many things are more expensive if you have credit issues - did you know that your car insurance can cost more becaues of low credit scores? Your car lease or purchase payments are likewise affected as are all credit cards plus some health insurance companies won't insure you, utiltiy companies may ask for a security deposit or for an increase, you might get turned down for a rental and it will hurt you if you're looking for a job and on and on and on.
Here's a question for you - how's your down payment? If you have a low credit score and a low downpayment, the best thing you can do is to wait a while and save up some more to increase your down payment. That's the fastest way to improve your mortgage picture.
The reason you're having a problem is because the guidelines were recently raised from 580 to 620 by the FHA which has the most liberal guidelines. Frankly, I think you should continue to work with the banker who's spending time helping you - the advice is correct. Work on raising your credit score and also work on trying to save a bit more for your down payment if you can. The real estate market is not changing quickly so you have the luxury of enough time to put yourself in a better position.