The bug
Endermen do not aggravate immediately after being stared at; instead they have to be stared at for 5 game ticks.
How to reproduce
Summon an enderman riding a minecart:
summon minecart ~ ~ ~ {Passengers: [{id: "minecraft:enderman"}]}
stand a few blocks away from the enderman, and move the cursor quickly from left to right, passing through the enderman's eyes.
If the enderman doesn't aggravate, pass through its eyes again, and move a bit slower than the previous time each time.
Notice that when then cursur moves quickly, the enderman doesn't aggravate; but when it passes slow enough, the enderman aggravates.
Comments

I can confirm this issue.