Smell, excessive noise, and truck traffic will likely become factors.
It is a known fact that neighborhoods located in a proximity to rail roads, airports, express ways, landfills, industrial areas, etc. do not fair well when it comes to maintainiting their value. This happens largely because these occurances have a serious impact on the quality of life in these locations.
If the asphalt plant is isolated and far enough from the residential areas it may have little effect. On the other hand, if it's sight, sound, or smell are impacted residents may likely have reason for concern.