Villager stuck/spins on scaffolding to reach work block

I cured a couple of villagers and I gave them various work spots around my house but they seem to be interested in my grindstone and barrels that are underground and only separated by scaffolding(I use the scaffolding as an "elevator"). They just get on the scaffolding and spin around rapidly fast instead of finding a closer work spot. I attached screenshots showing the layout.