The Agent has a fiduciary responsibility to their client. If you signed a "Buyer Broker" agreement with the agent, then you hired that agent and that agent works for you and has a fiduciary responsibility to you.
A real estate agent who is hired by the Seller to market and find a buyer for their property also has a responsibility to the buyer to tell the truth, to disclose everything they know about the property and to treat the buyer fairly. But they do not owe you any allegiance. their goal is to get the job done for their employer, the Seller.
In California, an agent can represent both the Seller and the Buyer. That is called dual agency. Every agent is obligated by law to disclose who they work for as soon as possible after first meeting. The State of CA requires each agent to present an Agency Disclosure to each prospective buyer and seller explaining this.
I am concerned about a Broker who would say such a thing to you. The Broker is ultimately responsible for the actions of the agents working under his license; and has the utmost of supervisory responsibility for them and their actions.
If you'd like to email me the details of your issue with the agent , I'd be please to review and at least give you some direction.
By way of qualifying, I have been in business for 26 years, and own and run a Real Estate Transaction & Risk Management company. In short we are the people who handle the paperwork, coordinate the transaction process, making sure contracts, and disclosures are correct, etc... and keeping everybody out of jail. E Mail to: email@example.com