What exactly do you mean by "hurt?"
Every time you miss a payment, it dings your credit history. If there's value in the home, they could foreclose, but they'd have to satisfy the 1st and then sell the house to recapture their cash. They could sue you for the difference, but the law, as interpreted only by an attorney, would be the determining factor. Anyone can sue anyone.
So, I'm not sure what your definition of "hurt" is.