Villager cant correctly get their ways to their house

At night some villagers will get stuck at the corner of the houses. Only when I destroy the wall ,they can get into the house and sleep.