Rude? absolutely not...but understand that are other ways of dealing with this issue as well. Simply shutting a window in their presence should send an unspoken message that you are having an issue. Any individual sensitive to the needs of others will get this loud and clear.
Considering placing a fan in the window that blows the air outward should make a difference as well.
Above all, advocate for yourself but do so in a diplomatic and sensitive manner that takes into consideration their needs as well.