I can appreciate your desire to do the right thing by this agent and be sure you are not acting inappropriately. I commend you for seeking out guidance. The bottom line is that this process is about meeting YOUR needs. Being you are out of area and taking time to travel, your schedule and attention need be considered. You need to choose an agent that meshes well with your expectations. That is something you need to be upfront about. Also, generally speaking... the tone of the first couple of meetings set the bigger picture of how your buying experience will go. If you are running into troubles at the first meeting... typically that would be a sign of future difficulties to come. However, all this being said... you MUST be honest with the first agent and phone or email them to inform them you have chosen to utilize the services of another agent. Be kind. Compliment the things they did well and be upfront about why you are chosing to take your business elsewhere. The best feedback we get in our business is from buyers and sellers that got away! Help that person improve their business for future clients. By being honest with them you will feel better about terminating your relationship. No one likes to be rejected, it may be difficult to do but it gives closure for all involved. Hopefully the agent will respond PROFESSIONALLY with apology and wish you well!!! If they don't, then you just got another sign that you made the right choice.
If you are not comfortable with an agent, you have the absolute right to switch and use someone else. Just don't do this after the agent has spent a lot of time with you. This isn't really fear to the agent. If they take you out and show you homes and you have been working with them for 3 weeks already, changing agents just because the initial one wasn't available at a certain time isn't very fair. Try and decide after the first initial visit if you are comfortable with them. If not, move on.
One other thing, don't use multiple agents. Many people think that this is beneficial to them, but it's the exact opposite. We as agents need for our clients to be just as loyal to us as we are to them. If an agent even thinks that you are working with someone else too, they will usually not be as loyal to you and spend time on your situation. Only one buyer's agent is going to get paid in the end. If you find an agent you are comfortable with, stick with them and be a loyal client.
If you have spent a lot of time with the initial agent, you could always ask the second agent to pay a referral fee to the initial agent at the successful close of escrow. This is if you really feel bad about skipping out on the first agent. If you don't think they were helpful to begin with, then just move on.