Server performance issues since 1.16.1

Was the debug profile run when the server was slow, or when it was just started? Also, when you look at the resource monitor of your server, is there any specific outlier? (Like a high CPU usage or RAM usage, or anything similar)

Hey,

So this was started while the players were on their servers and while it was skipping 5,000 - 20,000ms consistently.

Node usage is about 40% overall for that report above.

I've got another report if it helps:
CPU: Xeon E3-1245 v5 @ 40 - 45% ish (overall)

https://pastebin.com/mpswZHc5
ID 350809

When you ran the debug report, it should have also generated a .zip file, could you please attach that if it's there?

Hey,

Here you go:

Let me know if you need any extra data or more samples from other servers 🙂

Hi

I have this issue as well on my server (Core i7-3770 CPU @ 3.40GHz, 4 core 8 threads, 16 GB RAM), OS: Debian 9.
The save I use was created in 1.15.2.

I can provide logs, debug txt and a crash report. The crash report was with 1.15.2 datapacks installed, so I disabled them for the next start of the server. The issues still persisted.
Is there a way to upload logs without sharing them with all bug tracker users? I consider a lot of data in there as "sensible information".

But to further answer some of @unknown's questions:
The issue starts happening almost immediately after joining. I fly around a little with my elytra and the issues start happening after some seconds.
I monitored the CPU usage with top and interestingly it didn't spike, but it dropped significantly, often below 10% usage.
Next thing I monitored was Disk IO (with iotop).
For some reason minecraft 1.16 seems to be limited to around 250K/s DISK WRITE. My raid can usually handle about 40M/s, which isn't blazing fast by today's standards, but still enough for minecraft in the past. Another interesting part is, that there's 0 Read IO, while the writer is doing something. So I am starting to fly and chunks just don't load anymore. Then the "can't keep up" messages start to appear.
It seems that the chunk writer of 1.16 is blocking read access and does some serious funk in my world.
When I shut down the server, the writer still persists for quite some time. It looks like it's rapidly building some kind of chunk write queue and then slowly writes that to disk.

So just out of curiosity I flew the same route in my map, but in 1.15 (and with a restored backup of my world).
There's also a some IO happening on the same route, but lots of reads (chunk loading), far less writes, and no blocking of the chunk loading.

As I said, if needed I can provide lots of details, or a world download, or maybe a screenshare session. But I'd like to keep my data as private as possible.

So I was just testing again:

This is happening while just sorting my inventory to chests, I'm not even flying around, but when I try chunks don't load and these appear again:

Interestingly the write IO workers persist minutes after disconnecting from the server.

Hello, I'm just chiming in on this bug report to also note that this has been impacting me on the Raspberry Pi (ARM CPU)/Docker end of things. My server was running wonderfully until the last few weeks or so. The 1.152 and 1.16 update effectively made the server unusable, with the system being unable to keep up to the point where the watchdog forcefully steps in and restarts the server due to inactivity. This behavior seems to be consistent across any world file.

I can try to gather some logs and info, but it seems like the CPU/RAM/IO isn't particularly stressed so it's been really difficult to nail down what's wrong.

Hardware: Raspberry Pi 4B - 4GB RAM

OS: Raspberry Pi OS

Platform: OpenMediaVault 5 / Docker / Portainer

Container info: [itzg/minecraft-server|https://hub.docker.com/r/itzg/minecraft-server], 2GB RAM allocation, version 1.16.1 (was also an issue in 1.15.2)

I've attached a few crash reports. The 6/17/20 one is from 1.15.2, and the 6/26/20 ones are from 1.16.1.

Like what others have stated, the server is in "can't keep up!" mode before anyone even joins. If I join (I'm the only one who ever uses the server), the "can't keep up!" messages get more and more severe. Below are a few screenshots of the logs:

When the server starts up this is the state of htop (See file 2020-06-28 ServerStartCantKeepUp.txt for logs during this time):

When I walk a fair distance, the server starts to stutter, lag, and ultimately crash. Here's htop when that happens (See file 2020-06-28 ServerLagsAndCrashes.txt for logs during this time):

Hi, i observe the same performance issues. My hard drive writes between 250-300kb/s. No reading while writing. However, this behavior can also be observe when no player is online. In the past of Months, i havn't perfomance problems with different Server Versions, only with Version 1.16.1.

Java Edition
Server: 1.16.1
No mods, no crash reports
Synology NAS DS218+ (10GB RAM); Docker with 2 GB RAM for 3 players; Java Version 8u212

When I posted on Saturday I also tried Spigot for 1.16.1 and the bug was also present in their release.

Today I tried Paper for 1.16.1 and they seem to handle disk IO differently than vanilla/spigot. This bug isn't present in their 1.16.1 server (at the time of posting here it's release #21 for Minecraft 1.16.1).

I hope I don't break any rules by providing a workaround that relies on software by a non-Mojang party. But I wanted to at least share my knowledge with others who can't upgrade their servers to 1.16.1 due to this issue.

If you have this issue – please check whether sync-chunk-writes is disabled or enabled. If you're on linux, try disabling it and check if it's better. Linux is known to not handle sync chunk writes very efficiently, see MC-177542.

There seems to be a similar issue with Bedrock Edition servers on Linux - bug report here: BDS-2574

Apparently it also affects containers.

There's some suggestion this has existed since at least bedrock v1.14, but I think people are only really noticing it in large numbers now that performance has degraded to the point it has become unplayable.

Good morning, I gave the above suggestion a go (my specs are in a previous comment):

• Stopped container

• I set sync-chunk-updates to false in server.properties

• Started container, I let it fully load up

• <Server crashed upon connecting for the first time, world never loaded any chunks>

• After waiting for the server to restart, I was able to join and walk around, but my character could not eat food, and breaking blocks took at least 3 attempts, otherwise the blocks would come right back

• Logs were/are littered with "can't keep up! Is the server overloaded?" with the MS in the 5 digit ranges. Also some instances of "moved too quickly!"

Conclusion: setting sync-chunk-updates to false....I guess did something to help, but the performance still isn't nearly close to acceptable.

this is also happening on my server 1.16.1 release and i am up to 119% cpu with no one on the server after a reboot if someone goes onto the server it can jump to 135% , my host did some checks and they seam to think its to do with hoppers again. however i removed all hoppers from the world and it still did it , even though the logs blamed hoppers still

Checking in, the exact same problem is still present in 1.16.2 for me. Nothing has improved. Logs and crash report attached.

[17:35:48] [Server thread/INFO]: Preparing spawn area: 0%
 [17:35:48] [Server thread/INFO]: Preparing spawn area: 0%
 [17:35:48] [Server thread/INFO]: Preparing spawn area: 0%
 [17:35:48] [Server thread/INFO]: Preparing spawn area: 0%
 [17:35:48] [Server thread/INFO]: Preparing spawn area: 0%
 [17:35:48] [Server thread/INFO]: Preparing spawn area: 0%
 [17:35:49] [Server thread/INFO]: Preparing spawn area: 84%
 [17:35:50] [Server thread/INFO]: Preparing spawn area: 86%
 [17:35:52] [Server thread/INFO]: Preparing spawn area: 88%
 [17:35:52] [Server thread/INFO]: Preparing spawn area: 88%
 [17:35:52] [Server thread/INFO]: Preparing spawn area: 88%
 [17:35:52] [Server thread/INFO]: Preparing spawn area: 88%
 [17:35:52] [Server thread/INFO]: Preparing spawn area: 88%
 [17:35:53] [Server thread/INFO]: Preparing spawn area: 96%
 [17:35:53] [Server thread/INFO]: Preparing spawn area: 96%
 [17:35:53] [Server thread/INFO]: Preparing spawn area: 96%
 [17:35:53] [Server thread/INFO]: Time elapsed: 45606 ms
 [17:35:53] [Server thread/INFO]: Done (46.119s)! For help, type "help"
 [17:35:53] [Server thread/INFO]: Starting remote control listener
 [17:35:53] [Server thread/INFO]: Thread RCON Listener started
 [17:35:54] [Server thread/INFO]: RCON running on 0.0.0.0:25575
 [17:35:55] [User Authenticator #1/INFO]: UUID of player <redacted> is <redacted>
 [17:35:57] [Server thread/INFO]: <redacted>[/x.x.x.x:60365] logged in with entity id 27 at (-154.70743812181644, 69.0, -7
64.5085599943443)
 [17:35:58] [Server thread/INFO]: <redacted> joined the game
 [17:35:59] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5394ms or 107 ticks behind
 [17:36:15] [Server thread/INFO]: [<redacted>: Teleported <redacted> to -67.5, 67.0, 51.5]
 [17:37:05] [Server Watchdog/FATAL]: A single server tick took 60.00 seconds (should be max 0.05)
 [17:37:05] [Server Watchdog/FATAL]: Considering it to be crashed, server will forcibly shutdown.
 [17:37:06] [Server Watchdog/ERROR]: This crash report has been saved to: /data/./crash-reports/crash-2020-08-14_17.37.0
6-server.txt

We have fixed the major causes of this issue. If you still have issues then create a new bug report with repro steps for that issue.