The Sellers have every right to not only remove the keys from the lockbox but to also remove the lockbox all together. Entry to the home is coordinated between you, your agent, the listing agent and the sellers.
At this point all entry to the home must be coordinated between the Realtor representing the Seller and your Realtor. This is not unusual. At no time during the purchase process is it permitted for you to enter without being accompanied by your Realtor.
They probably do not want anyone just showing up and going in the house including previous agents who may have the combo. Have your agent coordinate your inspections and final walk through to gain access.
Ask your Realtor how you will receive the keys. Sometimes the Realtor has to get them from the listing agent. Then they get turned over to you after the closing has taken place and the funds have been cleared.