Yes they can ask for more as the lease is changing now, even though your brother and mother had lived there for so long.
Maybe its time to consider a purchase? You have so many options available and down payment assistance programs that can help you. Connect with me for more info....I know, every agent is probebly saying that, I don't read past the my answer but Im happy to advise either way.
Depending on your brothers credit, you can even purchase together, or he can be your roommate & pay rent to you.
The total deposit (it doesn't matter if it is a cleaning deposit, security deposit, key deposit, or a combination of deposits) charged by the landlord can not exceed 2x the amount of the monthly rent (unless the property is rented furnished then they can charge 3x or if you have a waterbed, there can be an extra amount equal to a half months rent).
You may also want to speak with your local tenant's rights group if you feel that you are being treated unfairly.
Best of luck.