Usually the wind is from the North, a nice breeze, and there is no noticeable smell.
Since there are significantly fewer people and cars, the air is a lot cleaner than San Jose.
I haven't sold a home in Gilroy where the smelliness of a home/neighborhood was an issue, but I have sold homes in certain areas of Milpitas and San Jose where those kinds of issues came up. The best way for you to figure out the "smelliness" of a property and/or neighborhood is to go there and smell for yourself. :) Only you can decide if the smell is noticeable, bearable, good, fine, or bad. If it helps, you could also talk to people who live in the neighborhood for their opinions. However, the smelliness is completely subjective, and only you know what your limits are.
Please consult with a trusted Realtor, who is knowledgeable of Gilroy and the neighborhoods you are looking to buy in.
Feel free to contact me with any questions/etc. regarding a property you are interested in! I can help you.
Best of luck,