I know this 21043 zipcode very well. I live here and sell all kind of properties here. You would be safe in most of the area!! Pay attention to the area and look at schools. 3 High Schools that are all over ranking wise!! So, be in the right place.
Everywhere else, I would say yes.
Mostly because town houses are often being used as rental property by the owners, and when it comes to rental, their is no limit either way. (+/-).
If there are more town houses then SFR, then it's worse.
If there are town houses with condo association, then it is the worse. A lot of condo associations are delinquent and because of that, it's hard to find lender to purchase such town houses. So, prices go down a lot quicker and pushes the market down. ( Discuss with a Realtor to under stand this in detail.
Now, with all the above possibilities, you should look at the possibilities closely and figure out what is best. Hard to say if any of the above issues will come up, that's why discuss with a Real-estate professional about all the possibilities.
I am sure, others will have different opinion.