I was in a situation like this, i can't give you advise online what to do as it's not exactly orthodox.
Example: what type of noise is it? Do you hear people screaming or possibly fighting?
but there are ways to get rid of them. This is why I never rent apartments or duplexes ever again in my life!!!!
My best advise to you is I would simply move out.
You won't get your deposit back though. You would have to chalk it up as a lost.
I broke multiple leases, never had a problem but that was due to other problems beyond anyone's control such as the association was creating problems and trying to extort money out of the tenant. Example: charging $125 for parking permits for every visitor that parked in the parking lot.
Listen at the end of the day if the landlord threatens you, simply call his bluff.
what's he gonna do huh? it's probably gonna cost him more time and money to pursue you in court than it's worth!
In fact if he's willing to lose a 6 year tenant, I would threaten him that I'm leaving, there's some other stuff I would tell you but not on line.