The 1.14 snapshots have some severe performance problems. Servers fall behind continually, and users are regularly kicked. The first thing I noticed is that even when nobody is logged on to a server, 100% of one CPU sure is being used. As you'll see, this is a red herring, but it means that everyone else is stuck until this is implemented properly.
I profiled this, and I found that Thread::yield dominates the run time:
Tree Profile:
(t 100.0,s 7.9) java.lang.Thread::run
(t 92.1,s 0.3) net.minecraft.server.MinecraftServer::run
(t 87.9,s 87.9) java.lang.Thread::yield
(t 1.9,s 0.0) net.minecraft.server.MinecraftServer::aI only have obfuscated names, here's where that's called from in MinecraftServer.run():
@Override
public void run() {
try {
if (this.d()) {
this.Z = k.b();
this.n.a(new jf(this.E));
this.n.a(new pd.c("18w43c", 442));
this.a(this.n);
while (this.u) {
long l2 = k.b() - this.Z;
if (l2 > 2000L && this.Z - this.Q >= 15000L) {
long l3 = l2 / 50L;
h.warn("Can't keep up! Is the server overloaded? Running {}ms or {} ticks behind", (Object)l2, (Object)l3);
this.Z += l3 * 50L;
this.Q = this.Z;
}
this.Z += 50L;
this.a(this::aU);
while (this.aU()) {
Thread.yield();
}
this.P = true;
}
} else {
this.a((b)null);
}
}Specifically:
while (this.aU()) {
Thread.yield();
}There are multiple other places where Thread::yield is called from, including another place in MinecraftServer and in ty.java.
This may be decompiled code, but it looks intentional. Basically, it looks like a hack that was necessary to get a snapshot out in reasonable time. No negative judgement there; we all do cheap hacks to get alpha releases usable for others, and we're better off with having a snapshot than not!
Unfortunately, this makes it hard for anyone else to help with profiling the server, and as a result, we are unable to help the developers find the performance hotspots that their profiling tools miss.
This problem also makes it inadvisable or disallowed to run 1.14 snapshots in the cloud, as some VM hosts will kill CPU-hogging processes, and it also can cost more.
Therefore I urge the developers to please implement this properly sooner rather than later.
Linked issues
is duplicated by 9
MC-138816
Integrated / "Vanilla" Snapshot Server's CPU usage is 10 times higher than in 1.13.2
Resolved
MC-141694
Sustained high CPU on java server while doing nothing.
Resolved
MC-141895
server process 100% CPU usage even without players
Resolved
MC-143212
Server 100% cpu on new empty world
Resolved
MC-143854
Idle java server load 100% on snapshots
Resolved
relates to 2
Attachments
Comments 38
Whoops. My bad. I'll edit the bug description, because THAT is the place where it's hammering the CPU from.
18w44a hasn't done anything to help with this. Here's the top of the profile tree now:
(t 100.0,s 13.1) java.lang.Thread::run
(t 86.9,s 1.7) net.minecraft.server.MinecraftServer::run
(t 52.9,s 52.9) java.lang.Thread::yield
(t 14.0,s 0.0) net.minecraft.server.MinecraftServer::a
(t 14.0,s 0.0) ti::b
(t 14.0,s 0.0) net.minecraft.server.MinecraftServer::b
(t 13.8,s 0.0) ua::a
(t 13.7,s 0.0) tz::a
(t 13.5,s 2.1) tz::n
(t 3.4,s 2.9) azp::c
(t 0.5,s 0.0) java.util.TreeMap::get
(t 0.5,s 0.4) java.util.TreeMap::getEntry
(t 0.1,s 0.1) java.lang.String::compareTo
(t 2.2,s 0.0) tq::c
(t 1.2,s 0.0) java.util.concurrent.CompletableFuture::getNow
(t 1.2,s 1.2) java.util.concurrent.CompletableFuture::reportJoin
(t 0.6,s 0.0) com.mojang.datafixers.util.Either$Left::left
(t 0.6,s 0.5) java.util.Optional::of
(t 0.1,s 0.1) java.util.Optional::<init>
(t 0.4,s 0.4) com.mojang.datafixers.util.Either$Right::left
(t 2.2,s 0.2) azt::a
(t 0.8,s 0.6) java.util.Random::nextInt
(t 0.2,s 0.1) java.util.Random::next
(t 0.1,s 0.1) java.util.concurrent.atomic.AtomicLong::compareAndSet
CPU wasting time is still an issue with 18w46a:
Performances are worsened since this snapshot version, with players being kicked or server crashing upon exceeding max tick delay or being frequently overloaded even when no-one is online.
I updated the world I was running previously, same conditions inside a docker container from Pterodactyl and it is no longer idling at 100%. Idling below 10% here. Host OS is Ubuntu 18.04.1 LTS and container is Alpine
On my server machine (Ubuntu 18.04, 2x8 core Xeon) things are better but still not perfect with 19w11a. The machine is running one instance using 19w11a and two using 1.13.2. The newest server is idling around 16% of a CPU and while the others are idling at about 2%. While this is a great deal better than the 100% it was using until now, it would be great to get it back to where is was in previous releases. Hopefully it's just due to excessive debugging code in the pre-release version.
The released 1.14 server .jar is still affected (I do not know how to correlate the NNwNNa snapshots to the releases, sorry).
I host 4 minecraft server instances on an Arch Linux amd64 machine, and due to this issue I had to suspend hosting after I upgraded from 1.13.2 to 1.14.0. I can definitely see how one could get a nasty surprise when hosting this on a cloud service.
SHA-384 checksums here:
951dbcef3af5f952cafb9a89379aed2018d8eebe4e9ebf801bd03c44719931d8652ddc7dab83f0d26e2568e8add4b788 minecraft_server.1.13.2.jar
2e691ee6bc50f67b9ad2237d86c30e7394aba39d73e20fa43b535f1c74b01238a8724965fdb22c1b3f52501f94005fe6 minecraft_server.1.14.jar
The 1.14 server .jar file has been obtained from here.
Here is a commented screenshot from a graphical systems load indicator when stopping, then starting, and then stopping again the 4 service instances:
[media]The baseline idle CPU load can be seen pretty well.
The shown time window is 4 minutes (120px, 1 pixel = 2 seconds), so yeah the huge startup load is pretty bad too, I am quite sure 1.13.2 started up in a fraction of that time. This is not an embedded system or anything, it is an entry-level server, the CPU being an Intel Xeon E3-1225 v3 with 4 cores and 3.2 GHz having 32 GiB of memory.
Java runtime environment:
openjdk version "1.8.0_212"OpenJDK Runtime Environment (build 1.8.0_212-b01)OpenJDK 64-Bit Server VM (build 25.212-b01, mixed mode)
I can confirm that this bug is still in the latest stable 1.14 release. CPU load is at 100% when no players are logged in.
There's a similar ticket for versions after 19w12a: MC-146579
Please post updates there.
It appears
net/minecraft/server/MinecraftServer.run()callsThread.yield()as well while waiting for the next tick, according to this StackOverflow answer this might not be ideal either.