Despawning Villagers

In my world I had a lot of villagers who I moved to an underground enclosed space with torches everywhere and no possible way of escape for my villagers. But one day I noticed that one of my cartographers, librarians and fletchers were missing and they were all master level so they couldn't change profession. I looked everywhere, but I couldn't find them. I assumed this would be a one time thing and just bred some more villagers to make up for the losses. But 2 days later it happened again when I was missing my armourer and another cartographer. I knew they were missing, because I had bought armour from the armourer the previous day. I hoped it wouldn't happen again and bred some more villagers. I slept in my house. The next minecraft day I went down to buy something from both of my librarians, but when I got there, I noticed there was only one. Then I realised that 6 out of my remaining 13 villagers were missing. I was very perplexed as I saw all 13 just 5 minutes ago. I really didn't want to start a new world as I already put so much progress into this current world. So now I'm here writing this. Any help to what is going on here would be very much appreciated. Thanks and have a great day!

In short: When I check on my villagers some despawn every day.