Noted that this issue also happens if the entrance to the building/house has a dirt path block right in front, villagers wont go through the dirt path to access the building/house.
For replicating this just have a building/house with dirt path block in front of the doors or points of entry.
*edit: just rephrasing for better conveing what I have observed. Not stating this an easy fix or the whole issue.
Noted that this issue also happens if the entrance to the building/house has a dirt path block right in front, villagers wont go through the dirt path to access the building/house.
For replicating this just have a building/house with dirt path block in front of the doors or points of entry.
*edit: just rephrasing for better conveing what I have observed. Not stating this an easy fix or the whole issue.