Villager Uses Point Of Interest Block Even If It Is Underground

I was building a market stall with a Shepherd villager in it and hid the loom underground so it could still have it's profession, and somehow the villager uses the loom underground even thought it is not accessible. Please fix this, thanks.