Villagers slow to take jobs

Not entirely sure why but during work hours whenever I take away the lectern from a villager it takes 30+ seconds until they become unemployed. But when the lectern is put back it is instant to get a job.