Villagers accept far away workstations

When breeding villagers, some villagers accept a profession (e.g. a weaponsmith) even if there is no grindstone nearby. The only way for me to make him a 'blank' villager, is to remove every grindstone in a massive area, which doesn't seem right. This issue isn't just for weaponsmiths, it affects every profession.