Yes, it's legal for an owner to rent without some kind of agreement in writing. A written agreement (in real estate) makes the agreement enforceable. But it's just as legal either way.
The problem, however, is that the tenant doesn't have a written agreement establishing the intended disposition of the security deposit or pet deposit. The tenant can say that it was to be refunded. The landlord can claim something different--for example, that the two months was for first and last month's rent and therefore shouldn't be refunded. Or the landlord might say that the tenant moved out earlier than the oral agreement provided and that, therefore, he's retaining the security deposit as damages.
In that case, how do you establish who's telling the truth? Tip: Look for any evidence. For example, if the tenant wrote a check, did he/she note in the memo line that it was for a security deposit?
Check with a lawyer for further guidance.
As far as security deposits, no matter what type of tenant you are, whether you have a lease or not, security deposits are heavily regulated, a landlord can not just keep it without legal cause. Security deposit rules also mandate how much of a security deposit a landlord can accept, here in Nh for instance they can only collect 1 month security, they can not demand pet deposits, last months rent just your 1st months rent and 1 month security deposit. upon 30 days of moving out they have to retrun the deposit in full or have an itemized bill with receipts to repair any damage.
If you google NY landlord tenant law security deposits, it will pull up the the law in NY for you. There are sever penanlties for landlords who violate the law like triple damages in some cases.