Villagers changing profession

I was trying to make a fisher man villager. I bought an item from him trying to lock the profession. When i took the barrel away tough he changed the profession to mason (i had a stone cutter nearby). I tought nothing of it just a bit annoyed but then when i looked at him his mason xp was grown like i would’ve bought from him. I didn’t buy anything from him after he changed to mason but somehow when i tried to change it back to fisher man he was locked to mason even when i took the stone cutter away. Had to kill him and i was so annoyed when i had to make another villager. Please fix this. Thank you!