My suggestion would be to call the ex-broker, the Broker he had when he started working with you:
Tell the Broker that you want to still work with the Agent.
This should nip any problems in the bud.
Most Brokers are understand and do not want complication.
Good luck and may God bless