Disappearing Villagers

So I decided to start a villager trading hall, it was secure with the only way in and out being a water elevator, and a piston door activated by a lever guarded the entrance, it was well lit, and their were no ways that any mob could enter. I had librarians with really good books ( Mending, unbreaking, efficiency, etc.) I also had 5 cartographers, 3 farmers, and a fletcher all maxed out. I leave and go out to get sand in a desert around 1000 blocks away so I can smelt it into glass and sell. When I return, 3 villagers remain, all are unemployed and are of little use to me. I bred them, and have a decent amount of villagers again, I copied my world, and went on another trip for sand, and as I return i find that 2 villagers remain, the rest vanished. I haven’t had this problem in my past worlds, and have traveled 10s of thousands of blocks without them disappearing in said worlds. I poured a lot of time and effort into this and I’m quite annoyed at this glitch.