I moved 2 villagers from the village to my house, one defined his work as a librarian with trades, the other took the profession of Fisherman, because he took one of many Workstations in my house, but I deleted all the Workstations, but the villager does not change his profession and does not sleep.
I checked his brain memories with the command /data, they showed that his Job Site, and Home are in different places where i've put it and doesn't update if i move them.
With the librarian occurs something simillar but the difference is his Job Site don't change and don't sleep.
Is like they can't updated their memories but i don't know if my hardware affects someting.
If you need screenshots i can add them later.
Please check if this is still an issue for you in 1.16 Release Candidate 1 or later.