After making a raid farm in 1.14.3 that worked, the 1.14.4 update seems to have broken it. Pillagers will pathfind into the village *IF they have a direct route, otherwise they will never make their way towards the middle of the village, therefore never becoming agro'd on the villagers. They also, when having issues pathfinding, will not attack me when I go near them.
Please check if this is still an issue for you in 1.16 Release Candidate 1 or later.