Endermen accumulate in overworld spawn area and destroy the landscape

When playing on survival servers I have often noticed, that there are a lot of endermen in the spawn area. As an admin I occasionally just used /kill to kill all endermen that where holding a block.

The problem is, that when the endermen spawn, they pick up a block and don't despawn anymore. And then they move around and maybe put down their block and pick up a new one. This way you have a lot of random blocks everywhere, your builds degrade over time and there are entire ravines(the endermen accumulate especially in the arches below mountains, because they teleport there when it rains) with random holes in them. I think this is amplified in the spawn area in comparison to anywhere else, because the endermen don't even despawn normally if they aren't holding a block.

This is not quite a bug, because it's rather in the grey area between bug and feature. It's kind of just an unbalanced game mechanic, that makes building quite annoying. If this isn't considered a bug, feel free to remove it. I also submitted this as a feedback, if this rather belongs there