Villagers keep profession even when their workstation has been destroyed

The bug

I set out to make a villager trading hall and when I set up the job tables I got everyone to accept their job just fine but I did notice that when they would accept the job they would flicker their profession a lot before finally deciding on it. After setting everyone up with a table right in front of them I decided to make a villager have the specific trades I wanted however on breaking the table they would not change to jobless and would keep their profession.

Things I have tried:

1. Breaking any and all beds around to see if they would update

2. Killing any villagers that were jobless and re trying the whole table breaky thingy

3. Removing any blocks above the villager

4. Removing all job tables even the ones that I'm not dealing with which now every villager seams to be stuck in their profession

5. Opening up the space and giving them more room to see the table 

Once I removed all the blocks on top of the villagers I did notice that a lot of villagers who were librarians changed over to clerics which struck as odd because now I have more clerics than brewing stands.

When I did break the blocks I was left with 2 villagers that were jobless so I tried to change their profession to librarian and now they are also stuck with their profession.

I did some testing in creative and it seams if the villager has room to move around they don't become broken like how I'm describing. If this is the case and they do require movement to change and relearn jobs then that might just how this has to be made.