After some further investigation I was able to figure out the cause.
A villager seems to randomly pick a free job site block in the area, even if it can't pathfind to the target.
So if a unreachable block is assigned, the villager will ignore another one, even if it is right in front of it.
Further, it seems like re-assignment happens after some time, so eventually the reachable one will be chosen and the villager takes the job.
Seems like the obvious fix is to exclude unreachable job blocks from the selection.
After some further investigation I was able to figure out the cause.
A villager seems to randomly pick a free job site block in the area, even if it can't pathfind to the target.
So if a unreachable block is assigned, the villager will ignore another one, even if it is right in front of it.
Further, it seems like re-assignment happens after some time, so eventually the reachable one will be chosen and the villager takes the job.
Seems like the obvious fix is to exclude unreachable job blocks from the selection.