Villagers that have workstations do not revert back to unemployed ones when their workstation is destroyed

I was shuffling the trades of the librarian when it offered protection 3, so I bought it and didn't have use for the protection 3 anymore, so I wanted to shuffle the trades again, but when I broke the lectern, the librarian didn't go back to being an ordinary villager. The same goes for a Fletcher (as far as I know). When you trade with them, they won't go back to being a jobless villager, even if you break their workstation.