You may qualify for an FHA with a mid 600 credit score. 620 is really the minimum. Since you are $100k upside down what you can do is work with a mortgage broker who can assist you with the refinance and "buy" equity down bring $100k cash to the close and then you can refinance and cash out your brother. You then take over the property with the loan in your name, have him sign off the title and off you go.
The other way to do it, is have him sign off on the property, the mortgage will stay in his name and when you are able to through cash, and forced appreciation in the next 5 years you can refinance and get the loan off his name.
Those are really your only two options for a refinance. Good luck!