Server performance issues since 1.16.1

Was the debug profile run when the server was slow, or when it was just started? Also, when you look at the resource monitor of your server, is there any specific outlier? (Like a high CPU usage or RAM usage, or anything similar)

Hey,

So this was started while the players were on their servers and while it was skipping 5,000 - 20,000ms consistently.

Node usage is about 40% overall for that report above.

I've got another report if it helps:
CPU: Xeon E3-1245 v5 @ 40 - 45% ish (overall)

https://pastebin.com/mpswZHc5
ID 350809

When you ran the debug report, it should have also generated a .zip file, could you please attach that if it's there?

Hey,

Here you go:

Let me know if you need any extra data or more samples from other servers 🙂

Hi

I have this issue as well on my server (Core i7-3770 CPU @ 3.40GHz, 4 core 8 threads, 16 GB RAM), OS: Debian 9.
The save I use was created in 1.15.2.

I can provide logs, debug txt and a crash report. The crash report was with 1.15.2 datapacks installed, so I disabled them for the next start of the server. The issues still persisted.
Is there a way to upload logs without sharing them with all bug tracker users? I consider a lot of data in there as "sensible information".

But to further answer some of @unknown's questions:
The issue starts happening almost immediately after joining. I fly around a little with my elytra and the issues start happening after some seconds.
I monitored the CPU usage with top and interestingly it didn't spike, but it dropped significantly, often below 10% usage.
Next thing I monitored was Disk IO (with iotop).
For some reason minecraft 1.16 seems to be limited to around 250K/s DISK WRITE. My raid can usually handle about 40M/s, which isn't blazing fast by today's standards, but still enough for minecraft in the past. Another interesting part is, that there's 0 Read IO, while the writer is doing something. So I am starting to fly and chunks just don't load anymore. Then the "can't keep up" messages start to appear.
It seems that the chunk writer of 1.16 is blocking read access and does some serious funk in my world.
When I shut down the server, the writer still persists for quite some time. It looks like it's rapidly building some kind of chunk write queue and then slowly writes that to disk.

So just out of curiosity I flew the same route in my map, but in 1.15 (and with a restored backup of my world).
There's also a some IO happening on the same route, but lots of reads (chunk loading), far less writes, and no blocking of the chunk loading.

As I said, if needed I can provide lots of details, or a world download, or maybe a screenshare session. But I'd like to keep my data as private as possible.

There seems to be a similar issue with Bedrock Edition servers on Linux - bug report here: BDS-2574

Apparently it also affects containers.

There's some suggestion this has existed since at least bedrock v1.14, but I think people are only really noticing it in large numbers now that performance has degraded to the point it has become unplayable.

Good morning, I gave the above suggestion a go (my specs are in a previous comment):

• Stopped container

• I set sync-chunk-updates to false in server.properties

• Started container, I let it fully load up

• <Server crashed upon connecting for the first time, world never loaded any chunks>

• After waiting for the server to restart, I was able to join and walk around, but my character could not eat food, and breaking blocks took at least 3 attempts, otherwise the blocks would come right back

• Logs were/are littered with "can't keep up! Is the server overloaded?" with the MS in the 5 digit ranges. Also some instances of "moved too quickly!"

Conclusion: setting sync-chunk-updates to false....I guess did something to help, but the performance still isn't nearly close to acceptable.

this is also happening on my server 1.16.1 release and i am up to 119% cpu with no one on the server after a reboot if someone goes onto the server it can jump to 135% , my host did some checks and they seam to think its to do with hoppers again. however i removed all hoppers from the world and it still did it , even though the logs blamed hoppers still

Checking in, the exact same problem is still present in 1.16.2 for me. Nothing has improved. Logs and crash report attached.

[17:35:48] [Server thread/INFO]: Preparing spawn area: 0%
 [17:35:48] [Server thread/INFO]: Preparing spawn area: 0%
 [17:35:48] [Server thread/INFO]: Preparing spawn area: 0%
 [17:35:48] [Server thread/INFO]: Preparing spawn area: 0%
 [17:35:48] [Server thread/INFO]: Preparing spawn area: 0%
 [17:35:48] [Server thread/INFO]: Preparing spawn area: 0%
 [17:35:49] [Server thread/INFO]: Preparing spawn area: 84%
 [17:35:50] [Server thread/INFO]: Preparing spawn area: 86%
 [17:35:52] [Server thread/INFO]: Preparing spawn area: 88%
 [17:35:52] [Server thread/INFO]: Preparing spawn area: 88%
 [17:35:52] [Server thread/INFO]: Preparing spawn area: 88%
 [17:35:52] [Server thread/INFO]: Preparing spawn area: 88%
 [17:35:52] [Server thread/INFO]: Preparing spawn area: 88%
 [17:35:53] [Server thread/INFO]: Preparing spawn area: 96%
 [17:35:53] [Server thread/INFO]: Preparing spawn area: 96%
 [17:35:53] [Server thread/INFO]: Preparing spawn area: 96%
 [17:35:53] [Server thread/INFO]: Time elapsed: 45606 ms
 [17:35:53] [Server thread/INFO]: Done (46.119s)! For help, type "help"
 [17:35:53] [Server thread/INFO]: Starting remote control listener
 [17:35:53] [Server thread/INFO]: Thread RCON Listener started
 [17:35:54] [Server thread/INFO]: RCON running on 0.0.0.0:25575
 [17:35:55] [User Authenticator #1/INFO]: UUID of player <redacted> is <redacted>
 [17:35:57] [Server thread/INFO]: <redacted>[/x.x.x.x:60365] logged in with entity id 27 at (-154.70743812181644, 69.0, -7
64.5085599943443)
 [17:35:58] [Server thread/INFO]: <redacted> joined the game
 [17:35:59] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5394ms or 107 ticks behind
 [17:36:15] [Server thread/INFO]: [<redacted>: Teleported <redacted> to -67.5, 67.0, 51.5]
 [17:37:05] [Server Watchdog/FATAL]: A single server tick took 60.00 seconds (should be max 0.05)
 [17:37:05] [Server Watchdog/FATAL]: Considering it to be crashed, server will forcibly shutdown.
 [17:37:06] [Server Watchdog/ERROR]: This crash report has been saved to: /data/./crash-reports/crash-2020-08-14_17.37.0
6-server.txt

We have fixed the major causes of this issue. If you still have issues then create a new bug report with repro steps for that issue.