The problem is a certain chunk shown in this video:
https://www.youtube.com/watch?v=qsczEpgyyAg
Crossing the chunks borders in a positive X direction or a negative Z direction results in a short freeze, only reproducable on the server and not in the same world being played singleplayer. Only the person crossing is affected by the freeze.
And as shown in the video only this single chunk causes the problem.
Linked issues
Comments

Duplicate of MC-162253.