So far it seems like this only happens if you put your game on Peaceful. I went mining for coal to trade to my armorer (his profession was locked and there were 4 villagers) I came back up and there were two villagers. And this changed almost every time I relogged and changed difficulty intil it was stuck at two.
Related issues
Comments

We're tracking this issue as MCPE-21416, so this ticket is being resolved and linked as a duplicate.
If you would like to add a vote and any extra information to the main ticket it would be appreciated.
If you aren't already, please don't forget to use the search feature, the less time volunteers spend linking duplicates the more time we have to update new reports.
Voting on an existing report has a greater impact on getting the bugs most important to you fixed!
Quick Links:
📓 Issue Guidelines – 💬 Community Support – 📧 Customer Support – ✍️ Feedback and Suggestions – 📖 Game Wiki