Yep. If you didn't agree to pay them, and have nothing signed that says you'd pay them... they are S.O.L. (Sorry, Outta Luck).
You can (and should) refuse to pay. They should have had that discussion with you BEFORE they brought their client to view your rental... and should have had you sign a document stating that you would compensate them... and exactly how much you'd pay.