Villagers switch and take each others beds

I have made a mountain village that acts as a breeder during the day. To get to the point, it is required that the correct villagers go to their designated beds, but sometimes, especially when two villagers share a house, they will get up in the middle of the night and switch beds or find an underground house and stare at a villager as they sleep, even though a spare empty bed is right next to them. I think maybe their pathfinding also needs a little work? They won't sense beds next to them but they can sense already claimed beds that require climbing ladders to get to. Help?