In a mesa biome in a mine shaft i accidentally attack an ender man.
He made the angry attack sounds but never attacked me.
There were also illager raiders next to the ender man
he followed me around for an entire minecraft day, sometimes appearing right in front of me but never attacked.
I was unable to reproduce in 1.16.2 Pre-Release 1.