I attached a couple of crash reports and a leak suspects report from eclipse memory analyser of a heap dump I took during the severe lag. In particular for the crash-2019-05-09_02.14.23-server I was disconnected because of a StackOverflowError which makes me think it has something to do with memory use. In order to recreate these issues I was by myself on a local 1.14.1-pre1 vanilla server with 512MB of RAM just teleporting to random locations, waiting for all the chunks to load and then teleporting somewhere else (mob spawning enabled).
Each time I teleport to a new location all the old chunks should unload and be purged from memory when the new ones load, but instead performance progressively degrades. I was using the default render distance (10) which means 21*21 chunks = 336*336 blocks, times 256 for the world height = 28,901,376 total blocks. I'm not sure how chunks are stored in memory but the heap dump shows 206.5/500MB of memory being consumed by what I presume to be chunks leaking memory, given a reasonable estimate of ~2 bytes of storage for each block the memory usage for the chunk cache should be less than 60MB.
I noticed that in crash reports from 1.14, ServerChunkCache was reaching upwards of 12000 after only a few hours of playing with less than 10 players online. On 1.13 the ServerChunkCache (in a crash report from August 2018) was only 3480 with 30 players online and the server had been running for quite a few days. Could there be a memory leak here with unloaded chunks failing to be garbage collected?
This might relate to MC-151674