You can get started in this business on a part-time basis. Especially with the hours you outlined.
One thing that will be key, though, is to work with a partner who you trust to split deals with you. Some buyers are demanding and it's better to be able to tell your clients that you can get them into the house this afternoon but you just need them to take a look at the home with your "partner" because he/she is free and you're booked up. That way the customer knows you're there for them and working to get them into their home even in spite of the fact that other commitments in your life do exist.