I've locked them in cubicles now so it's not an issue for me anymore but, here's some screenshots in case it helps.
1. Villagers would walk between where i'm standing and the warped wood frame (with door filling the gap).
[media]The second hole would also contain a double door. Villagers would ignore the second door.
[media]This is the area where they would sleep.
[media]
It should be noted that they didn't have access to working stations yet. There were some on the level above which they may have been assigned to. They also seem to be trying to reach it as the place where they gather is below those and there was no village bell yet.
[media]
So in hindsight, there's a lot of things that could be causing this.
Can confirm.