I've trapped a Librarian ( he had no profession until I made a Lectern ) inside a house and there is no open gap or any way you can enter inside. But another villager spawns inside randomly when is time to gossip with one another. I may be 10 blocks away or 1000 blocks away it still happens when the house is out of my vision.
EDIT:
After further inspection, when it's time for the villagers to go to sleep, the villager sleeps in what i guess is his bed. It just happens to teleport him when entrance is blocked.
Seed: 1542930509585342543
Exact building: -182 / 68 / 247
Please provide a screenshot of what exactly you did with the F3 debug screen enabled.