Problem:
Our minecraft server is running slowly and using 100% CPU, even when no players are online.
What happened:
We built an ice road in the End.
While riding boats along this road, the server started to use a lot of CPU and run slowly. (Down to 1 tick per second!)
The lag continued after we returned to the main dimension.
After we all logged off, Minecraft continued to use 100% of the server's CPU until we restarted it. Our server normally uses very little CPU when no players are online.
Restarting the server fixes it. Returning to the end makes the problem start again.
I have attached profiler results. These were recorded when I was in the main dimension. Note that the End dimension is still at 91%. Most of this comes from 'chunkSource' and 'tickPending'.
Steps to reproduce:
Download our world here: https://www.dropbox.com/s/86sm43xd6cpg0xo/end-lag-test-world.zip?dl=0
Go to the End.
Teleport to -3853 106 0
Ride a boat back towards 0 0 0 (or just run and jump about 30 jumps)
You should notice some lag (i.e. boat position being reset every few seconds, or a slow server tick rate in the F3 debug info if running a local server) after a few seconds. The lag might come and go a few times each minute.
You can now use /debug start and /debug stop to record what is slow with the profiler.
If you return to the main dimension (/tp 0 64 0), you will see that the lag continues. The profiler will show that the lag is coming from DIM1 (the End).
Linked issues
Attachments
Comments

I've attached a crash report.
I now think the problem is connected to having waterfalls flowing into the void.
Visiting a waterfall (in the end) seems to cause the problem more quickly.
Note that once the lag starts, it continues even after moving away from the waterfall, leaving the End, and logging off.
Removing all waterfalls from the server seemed to stop the problem from happening.
Here are the locations of waterfalls in the attached world:-/tp
14930 108 0
-/tp4355 108 0
-/tp431 108 0
-/tp280 108 0/tp 104 47 10
Edit: removing the waterfalls didn't permanently stop the problem.

Can you please check in how far this applies to 1.14.4?

It seems to be fixed in 1.14.4!
I loaded up the same world and followed the same steps as before. I observed no lag while riding the ice road. The server never slowed down and the /debug stop file showed chunkSource as only 5% of the total.
On the server, please force a crash by pressing F3 + C for 10 seconds while in-game and attach the crash report (
[minecraft/crash-reports/crash-<DATE>-client.txt|http://hopper.minecraft.net/help/guides/finding-minecraft-data-folder]
) here.