Villagers disappearing then finding them all at natural spawn/first spawn.

It's happened on every world I create. I find a village, setup base. Everything's fine the villagers are all doing their thing then one day I'll get on and poof, all disappeared. I couldn't figure out why it kept happening. Then one day I strolled past the first spawn when I created the world and there they were. Just running around aimlessly. Dont think that's normal. It will happen in every village I enter.