Villagers Disappearing

For reasons unknown to me, my villagers are disappearing. They haven't been dying to zombies or any other source, and I've even switched to spectator mode with Hitboxes enabled so I could check all of the terrain. My only guess is that whenever I use my elytra to travel away, or perhaps when using a nether portal to switch dimensions, that this must be when the game loses track of them and they cease to exist. It is worth mentioning that this used to occur when converting a single player to a minecraft realm, but I haven't done that in ages.