Villagers Disappearing

Villagers have been disappearing from my village. One second they are there, and if I leave and come back there is some gone. It may only be villagers that have not been traded with since my master farmers and librarians have not disappeared but, I am not sure.