In the city of SLO, the windy, cooler areas generally lie nearest Los Osos Valley Rd, Laguna Lake, generally northwest. In the summer, these neighborhoods can be as much as 10 degrees cooler than downtown. That's one microclimate.
Another is higher on the hillsides, eastern side of town, generally thought of as the Flora St. corridor. This is probably the "banana belt" you are wondering about. In this neighborhood, summers are a bit cooler, there is a light breeze (not an annoying wind), and winters are more frost-free. You can grow bananas here (and they will fruit); also fuchsias, which don't do well in hotter parts of town. It's a very desirable area, and of course more expensive, by and large, than other neighborhoods.
I hope this helps!