I'm not a lawyer, so this isn't legal advice. For legal advice, see a lawyer. However . . .
First, it's unclear whether you mean a verbal agreement to list your home with an agent, or whether you mean you agreed to sell your home verbally.
In either case, though, a verbal agreement in real estate is legal but unenforceable. It sounds more as if you were discussing a listing agreement with an agent and may have agreed verbally to list it, but you didn't sign any paperwork. If that's the case--and, again, I'm not a lawyer--then you and the agent do not have an agreement and you aren't bound to use that agent in order to sell your home.
Hope that helps.