I have this issue literally every time I disconnect from my wife's world. It's been almost a year. The only thing that seems to fix the problem is quitting minecraft externally and restarting it every time, which is time consuming and rather obnoxious but dealing with it any other way makes playing with her untenable.
I've got this glitch in 1.16+ my villagers will stand at the back wall, or at gates if they can't reach the back wall, until morning.