I recommend that you eat crow and tell the listing agent you want independent representation. Hire an Exclusive Buyer Agent in your area to represent you. They will know how to handle it. Go to http://www.naeba.org for a referral.
I have buyers approach me all the time with this dilemma. My first call is always to the listing agent to learn about their relationship with the buyer...I'm not in the business of stealing another agent's clients. I find almost all listing agents understand this and will do what's best for the seller/client, which is to sell the house. If not, or if they have spent a great deal of time with the buyer, I always insist that they are compensated for their time. They can be compensated through the transaction if your deal goes through, or you can just write them a check for their time. $75-$100 is reasonable for each showing.
And you shouldn't feel badly about this. Buying a home is complicated and although 99% of Realtors are honest people with a lot of integrity, we are all working within a system that is flawed and that favors sellers. Good for you for knowing that you are entitled to proper, independent representation.
They failed to mention the listing agent demonstrated EXCEPTIONAL integrity by referring you to another agent in the office. Be very aware, all agents in the same office do compete with each other. The agent you use will defend and promote the position most benificial to you while the listing agent will battle to keep price up and terms favorable and contingencies to a minimum.
You are making this way more convuluted that it needs to be.