I spawned a villager next to a blast furnace, but it used a nearby grindstone. The villager was at the same Y level as the blast furnace, about 1 or 2 blocks away from it. The grindstone was much higher and farther. Neither the grindstone or the blast furnace was already claimed by another villager. Between the blast furnace and grindstone, there was a composter, also not being used by a villager.
Please check if this is still an issue for you in 1.16 Release Candidate 1 or later.