Villagers not losing their job when their block is destroyed

I am trying to change the trades with some of my villagers that just grew up into adult villagers and attached to the nearby work stations, but when I destroyed the nearby workstation and all other similar ones, he didn't lose his job. I first thought I was just missing one or just had one that I was unaware of, so I killed that villager and got a new one, and it happened again. I've tried this with multiple new villagers and I keep getting the same result as well as with different workstation. For example, the first one turned into a stone mason when I wanted him to be a librarian. I remembered that I didn't take down my stone cutter from building, so I went and destroyed that and he didn't lose his job, and then I went and took down the other one or two that were at my place and he still didn't and then this new one is doing it as a librarian and won't lose his job so I can switch it for a different trade.