The villager will not move after his workbench is destroyed

What I expected to happen was...:
I thought when I destroy the workbench of a villager and then put it to a new place, the villager will move to the new place where the workbench at.

What actually happened was...:
However When I destroyed and rebuilt the workbenches of the villagers who had already traded with them.

A villager who had never been granted a job would reclaim the workbench and leave the former employed villager standing in place, unable to move

Steps to Reproduce:
1. Spawn 4 unemployed villager A, B, C & D.
2. Put 3 workbench (lectern) in front of them and let them reclaim the workbench.
3. Now there are 3 villagers (A B C) has been employed for a job, One villager (D) remains unemployed.

4. Trade with A B C, make sure they will not lose there job.

5. Destroy any of the three workbenches of villagers A, B, and C, and it stands to reason that they will not lose their jobs

6. Rebuild a same workbench (lectern). A strange thing happened: Villager D will occupy the new workbench. This will cause one of the three workbenches of villagers A, B, and C, standing in place, unable to move.

P.S

*Whether it's a bell ringing or sunset, they don't move