Villagers wont lose their profession

I was making a villager trading hall using a design where I can destroy the job site block so they lose their profession and when I place it back, they have different trades. When I was testing it, I spawned a villager and placed the job site block. As normal, it turned into that profession. When I destroyed the block, he kept that profession!