Villagers become unemployed when next to a workstation/ not restocking over time

I have a villager trading hall. The villagers are locked in a 1 block cube with a workstation 1 block away to restock. The issue is that when they are no longer looking at me the villagers that are novice become unemployed and the villagers that are above that refuse to restock after some time passes. The only way I am able to fix this is by letting them walk around and claim another workstation.