If you're unhappy with your agent's service, then you should discuss that with your agent. However, if you're not getting anywhere with that, then you should approach his/her broker.
Although you could go direct with the listing agent, you probably don't want to do that, because s/he has the seller's interests in mind. Some agents/brokers will do dual-agency (the relationship you're describing), and others won't. I wouldn't recommend dual-agency relationships for a first-time buyer. For that matter, I probably wouldn't recommend it for many non-first-time buyers too. Yet, I've worked with some agents via dual-agency agreements (as an investor), and I didn't have any problems.