Good answers here already. Just to provide my two cents:
Very likely it's a privacy issue. In fact, some people don't list with agents at all because they don't want people driving past, tromping through their homes, etc. So they're receptive to lower offers from investors who will purchase with just one or two visits. It's true. And some people who "like to drive by the house" sometimes decide to "drop in" unannounced. It's bad enough to have people cruising past, slowing down, stopping, and walking around. It's even worse to have them knock on the door asking for a "quick peek" inside. And, for that reason, too, don't assume there's a "For Sale" sign at the site. You'd be surprised at the number of homes for sale without any signage.
Also, it's possible that the exterior of the house doesn't fully reflect the inside. Sure, sometimes the house has no curb appeal and you might not want it for that reason. But I've seen plenty of houses that look OK from the outside, but kind of small. But you get inside and there's far more room than you'd have thought. Or something that just looks ordinary from the outside--not bad, just ordinary--that's spectacular on the inside.
A secondary factor is that some agents want to rule out the casual, non-serious looker. There are plenty of folks who consider it enjoyable just to cruise around going into open houses, gazing at homes that are for sale, etc. Sure, they might buy if the price were right. But they're not really serious. Plus, at some point serious buyers will need an agent. And it really makes sense to involve an agent sooner, rather than later. So some agents view a person who just cruises by properties for sale as not serious enough to invest time in.
As for whether it's a disservice to the buyer: No. An agent is responsible to his/her client. Is it a disservice to the seller? Not if the seller has requested the address not be disclosed.