This is a confirmed problem in 1.16.1. I'm honestly surprised they haven't fixed this yet. I suggest they completely redo the way villagers pathfind so that they pathfind to the nearest block instead of blocks they cannot access. Hopefully this bug gets fixed soon in 1.16 because if not, trading halls will be broken forever 😞
This isn't the same issue as MC-148309