RE-EDIT: I was able to reproduce the issue. Wait for a pillager patrol to pass, when they see you they will just watch you before deciding if you're close enough to attack or not. If you retreat and some of them move on but others keep staring at you, change your mode to spectator and make your approach, most likely these pillagers in particular, those who kept staring as you moved away, will aggro on you when in spectator mode.
_______________________________________________________________________
Just as the title say. You find a pillager patrol, look at them in spectator mode and they will attempt, unsuccessfully, to shoot at you.
Some pillager do not react at all and just stand there doing nothing for hours.
EDIT: I was not able to recreate this weird behavior. Only a few pillagers "saw me" when I was in spectator mode (even through the ground) but it did not happen anymore.
I don't know how to delete the post so don't waste your time on it.
They don't give up either. I made my house in a vast plain and I moved very far with the pillager (like 100 blocks away I think) and I came back to my house super fast, changed to survival again and next thing I know, the guy was against my window with his crossbow charged and ready to shoot me in the face.
Was able to reproduce it 3 times with 3 different patrols.