Villagers do not realise that their job site block has been broken

I wanted to change a weaponsmith to a cartographer. I broke the grindstone and he was still a weaponsmith. I got a new villager. replaced the grindstone and now there was two weaponsmiths. I broke it, this time with them both right in front of it. Then, the new villager became unemployed but the original is forever a weaponsmith. I replaced the grindstone again, and he acts like nothing happened, is now able to restock again, etc. I think what is happening is if he is far away from the block, he does not notice if it is destroyed. You could fix this by making employed villages ALWAYS load their job site block, and instantly become unemployed if it is broken.