villagers keep despawning in multiplayer

i play with my friend on our private server and we imported or cured villagers to our homes and built a little village of our own, but they just keep disappearing for no reason but we know that they AREN'T dying