Heather12, Don gave you a good answer. I too grew up with a B&O Freight train 1 street away. It became my lullaby. I couldn't sleep some nights without hearing it. The first home I owned had a trolley within 1 street away as well. So Don is correct you will get used to it.
We have an area nearby where people tend to pay more for the home with train stations nearby because they are taking the train into Philadelphia every work day. So it really comes down to choice for you and your family. However, have your representative to do a full market analysis between homes within a mile of this Railroad and non railroad locations. The properties closer to the noise usually sell for less, just make sure it is not significantly less, because if you re-sell the property years later will you be facing the same situation as this seller?
Best of luck to you,