Happened to me too, stopped happening once I gave the villager the tag NoAI:1. Don't know if you are talking about a villager summoned using /summon but I am.
Edit: command to replicate
/summon minecraft:villager ~ ~ ~ {VillagerData:{level:5,profession:"minecraft:weaponsmith",type:"minecraft:savanna"},Offers:{Recipes:[{rewardExp:0b,maxUses:9999999,uses:1,xp:0,priceMultiplier:1f,buy:{id:"minecraft:iron_axe",Count:1b},buyB:{id:"minecraft:dirt",Count:1b},sell:{id:"minecraft:iron_sword",Count:1b}}]}}
Cilan Dent
Happened to me too, stopped happening once I gave the villager the tag NoAI:1. Don't know if you are talking about a villager summoned using /summon but I am.
Edit: command to replicate
/summon minecraft:villager ~ ~ ~ {VillagerData:{level:5,profession:"minecraft:weaponsmith",type:"minecraft:savanna"},Offers:{Recipes:[{rewardExp:0b,maxUses:9999999,uses:1,xp:0,priceMultiplier:1f,buy:{id:"minecraft:iron_axe",Count:1b},buyB:{id:"minecraft:dirt",Count:1b},sell:{id:"minecraft:iron_sword",Count:1b}}]}}