VIllager Trade UI immediately closes when a villager has iron, gold, emerald and diamond badges

Right-clicking a leveled up villagers (Trading/summon command) will automatically close the trade UI. This only happens when they is no POI (Jobsite/Workstation block) nearby. Sometimes the trade UI is displayed when right-clicking again.

How to reproduce

1. Spawn villager

2. Put a workstation block

3. trade with the villager once

4. Destroy the workstation block

5. Trade with the villager again