Mindy - there is, indeed, a world of difference between 1 to 4 unit properties and 5+ unit properties. The financing certainly is different, as is insurance, etc. There are also additional government regulations that apply to managing a 5+ unit property.
But I wouldn't say that there are diminishing returns from properties that have more units. The opposite is actually true. The only question is whether or not you have the investor skills to swim in the deep end of the pool. Sometimes it's best to swim where your feet touch the bottom until you gain confidence in your ability to stay afloat.