Villagers are able to use Iron doors after player activates the door

I have my villagers trapped in a room using iron doors and stone buttons. Sometimes when I leave, I hit the button, the door opens, I leave, the door closes. Then a villager will open the iron door and walk out. This is not a matter of the villager being close enough to where they are trying to escape but the door is getting closed on them. This is them coming up to the door a short time after it has closed and then opening it. This happened multiple times, one right after another. the villager approached the door and opened it. The stone buttons are placed one block high (lower portion of the door) to the right of the door.