The best thing to do would be to contact the Pound Ridge Recreation Department. Here is the link for contact information:
Keller Williams Realty Group
There are actually numerous swimmable lakes in Pound Ridge. They are, however, all private and you would need to be a homeowner on the lake or have lake rights in order to use any of them. Pound Ridge does have a town pool available to residents and their guests for which you pay a nominal fee for the season. The town park also has a skating pond for residents.