1) If he's on the deed, he has rights to the property
2) If he in some other way, in writing, forfeited his rights of access, then you can change the locks.
The easiest way would be to get something in writing that acknowledges that it is your private residence, and he is an owner in name only. A lease won't have enough teeth. If he's not amenable, you could buy out his rights and put the deed in your name only, or various other options.