The client does not accept any server side changes to the offhand slot while a chest and other non-player inventories are open. Client side changes like item swapping (F button), do still work properly.
Modded servers do not have the capability to revert the client's item swap to the offhand till the opened inventory is closed. More specifically, this creates a client/server desync which the server cannot fix till that chest is closed. For creative mode players, this desync fully alters their inventory due to the power given to that gamemode.
This video demonstrates the problem, recreated in a single player 1.16.3 world by having a command block alter the offhand slot on a timer. It is noted that the slot stops updating while the chest is open. And resumes updating when the chest is closed.
Related issues
Comments

Thank you for your report!
We're actually already tracking this issue in MC-87019, so this ticket is being resolved and linked as a duplicate.
If you would like to add a vote and any additional information to the main ticket it would be appreciated.
Please be sure to use the search feature before creating a ticket to see if the issue has already been reported.
Quick Links:
📓 Issue Guidelines – 🛠 Community Support – 📧 Customer Support – ✍️ Feedback and Suggestions – 💬 Mojira Discord