You did not view the unit with broker 1. Also it sounds like you you do not have an agreement to work with broker 1 exclusively, which would have put another twist on matters, so therefore you have absolutely no financial obligation to broker 1. Go ahead and move forward before you lose the unit to someone else.
I think you're saying you only went into the apt with broker 1. If so and if that broker was the one who introduced you to the owner and worked with you on the app, then you should only be paying broker 1. I have often met with a client after she had been out with another broker. Often she has seen so many places she can't remember the names or addresses of the building until we get to it. And if she has been in it, we just move on to the next one.