Sporadic desync on server where I can't open inventory

I'm playing survival on a private server. Every now and then I find that my inventory won't open, I can't interact with chests and other GUI items, I can't use rockets, when I break blocks they break instantly and don't drop anything. I can move but when I approach mobs they don't attack me and I don't take fall damage.

It seems to happen more frequently when I exit my underground base into open air. Recently it's happened every time I go outside. The base is under a zombie piglin farm so we have two giant nether portals above the base - not sure if that would cause an issue. There are also two beacons on the surface near the exit. There are only 3/4 plays on the server and apart from the zombie piglin farm there are no big projects causing lag (a couple of villager trading halls with around 30-40 villagers nearby).

I assume that my client has desynced from the server and none of the client inputs are received by the server. I think it's just a one-way block. I.e. my client is still receiving data from the server but the server is not receiving data from the client.

If I log out and log back in I am back where was when the issue started and none of the blocks I broke before are broken. I can sometimes access my inventory once or twice but then it desyncs again quite quickly. If I close the game and reload then sometimes the issue is resolved but not always. I usually have to reload several times and restart my PC. 

I'm told by the server admin that there are no connection issues or error messages at their end and the server reports stable connection with my client.