Villagers are sharing workstations, when the workstation is removed, the villagers remain as it was. I am on a server and re-logging does not help this issue. As shown in the picture, there is only 2 work stations.