If the client unloads and then again loads a chunk with big structues, it causes lag spikes.
The first time the chunk gets loaded there is no spike at all.
Video demonstrating it:
https://www.youtube.com/watch?v=S9m9ijpLekQ
Notice how it does not spike if i change the view distance.
Is it possible you could attach a copy of the world for testing this?