I don't know if the agent will get in trouble, but please give, at the very least a courtesy to the first agent and let them know you have moved on without them.
If you end up purchasing the home you visited with the first agent, and the 2nd agent represents you in the transaction, the 2nd agent should have to pay a referral fee to the first agent for procuring the home you found.
I have one question for you Hex, do you believe in Karma? If so, please do the correct thing. Life is too short.