When a villager was sleeping in a bed next to me, I saw that he would completely vanish when he should still be visible. You can see it happening in the screenshots where I am only moving a couple pixels.
This is happening in Singleplayer, Minecraft version 1.17.1 (Optifine pre-release), Java 16.0.1
That is not my issue. My issue is when I turn around and the villagers are on my left/right side. You should still be able to see the villager. Compare my screenshots vs screenshots on MC-146881.