Despawning Villagers

When I create a Survival World I find a Village and given, it has villagers. After I have stayed near that village for a while the villagers begin to slowly disappear and ultimately the village becomes deserted. Even if I don't destroy their bed or work station they still despawn slowly.
I have even tried breeding, but I cant breed fast enough to prevent despawning. Please Fix This.