Yes, you can. By agreement between the two of you, or by revocation from the principal.
Will the broker ask you for expenses/commission? That's a different question, as the exclusive right to sell protects the broker.
Although a listing agreement is a contract and therefore consists of promises to perform that are enforceable on both contracting parties, should one's reasons to cancel the agreement be justified by extenuating circumstances (destruction of the property, force majeure, health issues...) I don't see why the contract cannot be terminated in a friendly way.
If you signed an exclusive right to sell, go through your paperwork to see if you did check the Safety clause box, which technically is a negotiated period after the termination of a listing, and which allows the broker to be entitled to a commission.
Revocation of the agreement:
If the seller decides to rescind the agreement, he/she may be liable for commission, part of the commission, or broker's expenses.
Please discuss your decision to cancel with your agent ASAP. Be sure to kindly explain your reasons and I hope you'll come up to a friendly agreement.