mojira.dev
MC-149018

High Idle CPU usage on Server Edition (Minecraft 1.14 Release)

It seems when Minecraft 1.14 is idle, it still continues to consume CPU at a significantly higher rate than its predecessors like Minecraft 1.13.2

 

I've compared the 2 side by side from VisualVM on both Linux and Windows, but I cannot find the cause when I run a snapshot from CPU Sampler. Both results are same, but the CPU consumption is too high on 1.14.

 

I have included screenshots from VisualVM running both Minecraft 1.13.2 and Minecraft 1.14

 

1.13.2 - Fresh start, No players, Monsters on, Animals on. 1GB RAM Allocated

[media]

 

1.14: Fresh start, No players, Monsters on, Animals on. 1GB RAM Allocated

[media]

 

Results did not change when I turned off Animals and Monsters.

 

Additional notes: 
On Windows, the problem is  slightly intermittent... sometimes it might run smoothly and then its almost like something triggers it to go out of idle mode and start munching CPU.
On Linux the problem occurs 100% of the time.

 

A second issue (which could be related) is that Minecraft is consuming up the server thread by 100% nearly all the time. Players join and stay in a single area, but seems like the Server Thread is still doing something way abnormally high. (Appears to be working with chunks, even though chunks are loaded, perhaps a bug in chunk loading / looping?)

As a result, the server console repeatedly reports falling behind in ticks by 80ms to 2000ms. 

NOTE: I have found that the second issue is a  known duplicate: MC-138550

So this bug will now only focus on the first issue, which is high IDLE cpu usage. Minecraft server instance uses a considerable amount of CPU with no players in server. It is also a significant waste of earth resources considering how many instances of Minecraft 1.14 is running around the world.

 

 

 

 

 

Related issues

Attachments

Comments

migrated
[media][media][media][media][media][media][media][media][media][media][media][media][media][media][media]
migrated

Same issue on both Prerelease 5 and Release 1.14. Makes the server agonizingly unplayable. Kicks every few minutes, projectiles/actions in slow motion etc.

migrated

I'm also seeing this issue with 1.14. I'm seeing the issue on a Linode instance running Debian 9 with 1 CPU and 2 GB RAM. I don't have servers metrics as detailed as in the description of this post, but I'm also seeing 100% CPU utilization when no players are on the server. CPU utilization was about 5% when running 1.13.1.

migrated

Confirmed on my home server (MC 1.14 and 1.13.2 running via mineos in a jail under FreeNAS).  Idle difference is about 1% on 1.13.2 vs about 14% on 1.14.  It's hard to capture in the CPU data, but visually at least, a single chunk /fill destroy command with about 20 blocks of height (16x16 chunk 20 blocks high, crossing subchunks 2 blocks above, 2 below; ~ 46 ~ to ~15 65 ~15) definitely appears to peg the 1.14 server much harder than the 1.13.2 server ever did.  It's VERY noticeable.  I've added an image of my results on my (antique?) Q6600 FreeNAS server box.  I will note that 80%ish CPU time use with Java is effectively CPU pegged because of FreeNAS doing other normal things on the server.

 

migrated

I can confirm this bug on a Debian 9 machine.

On 24.4 at 4:00 the server was automatically upgraded to the latest version and since then, the idle CPU load increased.

[media]
migrated

Seeing similarly high idle CPU usage on 1.14.1 pre-release 1.

migrated

Vainilla 1.14.1 PRE1

 

Frist Load

[media]

 

SSecond Load

[media]

in 1.13.2 (3 seconds to load)

in 1.14.1 -pre1 (10 seconds to load)

 

migrated

Adding that I am getting this as well with server version 1.14 .jar for Linux.

Running Ubuntu Server 18.04 within Proxmox, physical hardware is an Athlon 5350 w/ 16GB DDR3-1600, VM has 4 CPUs and 4GB memory.

The server is completely un-playable since upgrading to 1.14.

CPU utilization with no players connected:

 

top - 20:12:46 up 16 days, 4:53, 1 user, load average: 0.41, 0.35, 0.32
Tasks: 156 total, 1 running, 100 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.4 us, 2.3 sy, 0.0 ni, 93.8 id, 0.2 wa, 0.0 hi, 0.1 si, 1.2 st
KiB Mem : 4039092 total, 207184 free, 1703752 used, 2128156 buff/cache
KiB Swap: 4038652 total, 3752188 free, 286464 used. 2048400 avail Mem

PID   USER     PR NI VIRT    RES     SHR  S %CPU %MEM TIME+     COMMAND
20687 ________ 20 0  5275452 1.266g 20044 S *35.3* 32.9 395:49.22 java
18247 ________ 20 0  42812   3776   3196  R 11.8 0.1  0:00.03   top

migrated

Test in 1.14.1-PRE 2

[media]

 

server process is passed between threads
and that thread is always 100%

migrated

Can confirm that this is still an issue as of 5/11/2019 on version 1.14. My server is consistently at 100% CPU usage and randomly kicks people. It also randomly has the ticking issue and warns about lagging behind.  "12.05 02:43:38 [Server] Server thread/WARN Can't keep up! Is the server overloaded? Running 2022ms or 40 ticks behind"  It usually gets even higher than that. Occasionally it'll get bad enough where it crashes and I have to wait a while before even attempting at trying to start it back up.

migrated

Still an issue on Minecraft 1.14.1 release

migrated

I used to host a 1.10 spigot server with a bit less than 1Gb of RAM and the server was running nicely. Recently updated this server in 1.14 vanilla and added 2 more gigs of RAM to my VPS and, well tons of issues:

  • At startup, world loading does nothing for about 50 seconds and then every message pops in the console.

  • Like everyone else, 100% constant CPU usage even with only 1 player connected, only 14% when nobody.

  • Random "Can't keep up..." messages in console.

  • Random "Moved too quickly..." messages in console.

  • Extremly poor and unstable TPS.

Generation is just pretty wrecked now, it takes way more longer than before and may result in server crashes...

For me, 1.14 sounded like an exciting update, but I'm really disappointed of the performance gate experienced by servers...

migrated

Same problem here on the 1.14 and 1.14.1 servers. High CPU all the time, running behind (almost always more than 2000ms), randomly kicking people. I was excited to play 1.14 with my friends but it's completely unplayable in multiplayer.

migrated

Just updated to 1.14.2 and with similar server specs. to above. 4 users on, just breaking blocks, building etc. Not running ticks behind, and CPU and RAM pegged at 100%.

With no users on, RAM 43% CPU 100% - an illager trying to embark on a 40k block journey?

We prayed for 1.14.1 and the bugs fixes, then 1.14.2 and still no better.

Just to also point out, having this game peg CPU’s at 100% drawing more power, out-putting more heat is going to add up, and will become a monetary cost as well as an environmental cost.

Less Minecraft hosting servers are allowing 1.14.x to be run due the lack of smooth game play with friends, plus they don’t appreciate the very discourteous yells, screams and criticism from their users asking for support. The game is at fault, not the hardware, the setup, or the network.

It is very unfair, and would be nice to see this fixed.

So many times in streams I've seen "I'm going to give up on Minecraft and wait till 1.15"

migrated

Exactly. I'm going to have to wait for 1.15 or until I can afford to have a 3,000$+ PC built. Whichever one of those comes first.

migrated

There are even no hosting plans so high-end on the market.
I currently run my 1.14.2 server on Xeon 4-cores, SSD HDD and 16 GB RAM with render distance as low as 6. And TPS with 12 players is terrible - only 11-12. The server seem to be playable only below 10 players...
Back to 1.13.2 I could easily handle 30+ players on 3-cores and 10 GB RAM with render distance 15 and TPS constant 20.
This is simply a kind of joke... Did they at least run this game even once on a real test server before releasing it? We have now 1.14.2 and it is still as bad as 1.14.

violine1101

The bug tracker is not a discussion forum. If you want to complain about Minecraft updates, please do that somewhere else. If you don't want to add anything constructive to this ticket, please don't comment. Off-topic comments will be removed.

migrated

Affects 1.14.2

No users logged in, CPU pegged at 100%.

migrated

For the record, I've been experiencing this problem in 1.14 thru 1.14.2 with OpenJDK 8u212 on FreeBSD (server consumes around 75% CPU when empty), though strangely I do not see it on Windows using Oracle JRE 8u211 (idle CPU usage quickly drops to zero) when using the exact same map and config.

When I performed the exact same tests with Minecraft 1.13.2, CPU usage dropped to around 1% on both platforms.

migrated

Quietust, that seems to align with some of my tests. I've had a hard time replicating sometimes on Windows. It does happen though, just not as frequent.

Perhaps its to do with OpenJDK and not supporting a certain sleep based command. If so, I'm sure Mojang could figure this one out quick enough and resolve it.

 

migrated

I hope this qualifies as 'constructive'. 😃

I made some additional tests.

1. On Ubuntu 16.04 LTS VM running with Openjdk 1.8u191 ~ 20-25 % CPU usage on idle. (hypervisor was idle Intel NUC5I7RYH)

2. I ran the same version of the server on Windows 10 host with Oracle OpenJDK 1.8u181, the server was idle very soon after the initialization

3. Tried the same version of Oracle JDK, on the Ubuntu 16.04, and had the ~20-25% CPU

4. Enabled jmx console on the Ubuntu server and connected to monitor from Windows 10 host. I've used the Samples to monitor for the CPU usage and found out that the majority of CPU time spent is inside "Server Thread", namely in java.util.concurrent.locks.LockSupport.parkNanos ()  In total over 90%

5. I repeated the same as 4 on Windows 10 laptop with I7, and majority of time was still in parkNanos(), but with way less CPU usage ~ 50% and way less time spend in total!

Suggestions to the developers:

  • How many nano-seconds are you actually passing to the parkNanos()?

  • Would it be possible to make this configurable?

  • Is there an alternative, since it appears this function hogs the CPU on Linux systems?

migrated

In addition I found an interesting article dissecting the implementation of parkNanos()

https://hazelcast.com/blog/locksupport-parknanos-under-the-hood-and-the-curious-case-of-parking/

 

migrated

I can reproduce this problem on one of my dedicated servers.

My two servers have nearly the same configuration:

Server A: Intel i7-4770K, 32 GB DDR4 2666 MHz, SSD.

Server B: Intel Xeon E-2146G, 64 GB DDR4 2666 MHz, SSD.

They are both using Debian 9, with the same kernel version and the same OpenJDK version :

$ uname -r
4.9.0-9-amd64
$ java -version
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-8u212-b03-2~deb9u1-b03)
OpenJDK 64-Bit Server VM (build 25.212-b03, mixed mode

I am running vanilla Minecraft, version 1.14.2, with the following command:

java -Xmx1024M -Xms1024M -jar minecraft_server.1.14.2.jar nogui

Server A: about 5% cpu usage when idle (0 players online, new world).

Server B: about 70% cpu usage when idle (0 players online, new world).

migrated

@Maxou: as a user with a same problem, I am curious about your configuration differences. IMHO: having 2 different machines with different symptoms is perfect for troubleshooting.

  1. Could you please check the content of following files: /proc/${PID}/timerslack_ns (you need to find out actual PID of the running minecraft server). e.g. if the PID is 8132.

    PID=8132 cat /proc/${PID}/timerslack_ns
  1. What else could be different between these machines? (e.g. kernel options)

  2. Do they have any running hypervisor on any of the machines?

  3. Do the processes run as regular accounts or root?

  4. Perhaps the output of the following can be compared?

    dpkg -l | sort
migrated

@momcilo:

timerslack_ns is 50000 by default on both machines, no difference.
The CPU usage of the Server B is reduced from 70% to about 5% if I try to increase this value, to 200000 or more.
There is no change on Server A, no matter what I do with the value,

The processes are not running as root, but I get the same results as root anyway.

I tried to install Oracle Java instead of OpenJDK, I get more or less the same results, with less CPU usage.

The packages installed are the same (both are apt-get upgraded) and there is no hypervisor running on either machine.
There is a KVM over IP that can be used to control the machines but I don't think that matters.

Honestly, I'm starting to think the problem comes from the Xeon E-2146G, maybe because it's managing the mutex/sleep in a different way.

migrated

I commented before but forgot to mention the hardware I'm using. I used to be on 1.14.0 but now am using 1.14.1 and am still having this same issue. The specs of the server host I'm using is:
NVMe SSD drives.
Intel Xeon E5-1650v4 processors with a clockspeed up to 4.0GHz
and 10GB of DDR4 ECC 2133 MHz RAM

Server on idle will jump between 50% - 100% and stay there. Then when people log on it consistently stays between 98%-100% and we get all the lovely "server can't keep up" and large ms ticks. It can only handle 4 people on max now otherwise it becomes completely unplayable. 

migrated

@Maxou I could not affect the timerslack_ns due to the old kernel on my 16.04, so I installed a new 18.04, mscs and rsynced the same world there.

Findings on hosting the server inside VM (allocated 2 cores + 4GB RAM)

  • cpu usage was still high on freshly installed machine, that had only minimum required to run the minecraft server + mscs

  • I set 200000ns for the process using the CPU, and CPU usage inside vm dropped to just 10-14% (from 25), however the hypervisor itself was still using 35% with all cores being set to at 2.5 to 3.1 GHz (I disabled the TurboBoost for troubleshooting). Attempts to affect the VBoxHeadless through the same mechanism did not provide any change

Clearly, setting default timer slack is not an option for the VM, so I tried to the hardware itself:

  • installed mscs, synced the workd again and ran it without modifying the timerslack_ns.

  • the cpu usage went closer to idle and is now 13-14%, with all cores having frequency below 1GHz (finally!) The temperature of the CPU finally dropped

  • I moved another world (lots of redstone), and this one was running at 25% (all cores ~1.01 GHz)

So far we know that:

  • issue affects VMs (at least VirtualBox based, but possibly others)

  • issue affects some of the CPUs and some are unaffected. Mine is modest 2 core (4 thread)  Intel Core i7-5557U

For me the issue is mitigated at the moment since I moved to the direct hardware hosting, but at this point we need the developers to look into it.

IMHO: 15-25% on 1.01 GHz for an empty server looks too much.

 

migrated

I bring slightly more information.

I just noticed that the clock on Server B (the one that isn't working properly) is running too slow.

minecraft@debian:~$ date && sleep 60 && date
Sun Jun 16 06:24:29 CEST 2019
Sun Jun 16 06:25:40 CEST 2019

It seems to be a common problem on virtual servers: https://serverfault.com/a/356934, but can also happen on dedicated CPUs.

Installing ntpd fixes the problem above.

minecraft@debian:~$ date && sleep 60 && date
Sun Jun 16 06:53:45 CEST 2019
Sun Jun 16 06:54:45 CEST 2019

No difference on Minecraft though, the server is still eating more than 50% CPU with 0 players, but I feel like this might be a good track.

migrated

Still experiencing this issue on latest release Minecraft 1.14.3 using Ubuntu 16.04. Tried various CPUs.

Reading up on some posts above, @momcilo found that Ubuntu 18 resolved the issue as long as you set the  timerslack_ns to 200,000 ? Could you advise what you did to set the timerslack on the instance? I'll do some tests in my own environment. Is there any way to do it on Ubuntu 16?

migrated

Minecraft: 1.14.3

I'm using a Xeon x5670 (6c/12t) @ 4.0Ghz, 24gb ddr3 1600 mhz. and i'm getting 20-25% cpu usage with the server idle, and at least 50% usage with a single user online, and it hiccups every few minutes with 2 users.  I'm running Ubuntu 18.04 LTS, using MineOS to manage the server (bare metal, not a vm, or under esxi, etc)  I have 8gb assigned to the java instance running the server.

migrated

Same here. Proxmox with Ubuntu 18.04.02 LTS Guest and one Minecraft 1.14.3 Server running (i7-3770 Host) it takes 25% of 2 Cores with no player connected.

migrated

Still a problem on 1.14.4 Pre-Release 1

Tested on following CPUs:

i7 7700k
E5-2687W
E5-2670 & E5-2680 
X5690
D-1541

Idles between 17% to 25% of a core with no players, no monsters, nothing but a empty void world.

All CPUs are on dedicated hardware with Ubuntu 16 and 18 (no virtual). 64GB minimum RAM. CPU set on high performance.

Getting a little out of hand. Has it been looked into at all?

migrated

Still happening on 1.14.4 pre-release 2.

Between 50% and 60% kernel CPU usage (red instead of green on htop), 0 players online, new world.

[media]

Here is a /debug report, just in case it can be useful:

[media][media]

 

migrated

Can confirm that setting `timerslack_ns` to `200000` or higher makes the CPU usage drop.

I guess this can be used as a workaround until the underlying issues gets resolved.

migrated

@rafael ristovski - Is it possible to make this adjustment in ubuntu 16? If not, what OS did you use and what steps did you take to achieve this?

 

migrated

Once you find the PID (process id) of the server, echo "500000" > /proc/PIDHERE/timerslack_ns. Do note this requires a root shell. You have to do this every time after you start the server.

migrated

Still occurs in 1.14.4-pre5 on FreeBSD 11.2 - empty server CPU usage is around 75%, and a /debug profile report just says that it's managing 20tps and that 94.88% of the overall time is being spent inside nextTickWait.

Sadly, FreeBSD does not appear to have "/proc/PID/timerslack_ns" (and I can't find anything equivalent elsewhere), so I'm stuck without a workaround.

migrated

Not fixed in 1.14.4-pre6

[media]

Loaded up 1.14.4-pre6 with a brand new world, and no one logged in. The blip down from 100 is where I stopped the previous version and switched over to 1.14.4-pre6. The CPU jumped back up to 100%. 1.13 idled at or below 20%.

Environment:

Linode Instance
2 CPUs, 4 GB RAM
Debian 9

migrated

I'm seeing about  20% improvement at most, if that.

Where it was idling at 7.5% before (1.14.3).. now its 6% (1.14.4 Pre-6) on E5-2680 Intel CPU, Ubuntu 16 and similar ratios for other CPUs.

as a benchmark,  1.13 idles at 0.5% - 1.2% on the same CPU / OS

 

I think its got something to do with parknonos being used as a sleep/park function in 1.14 instead of the previous sleep in 1.13 and under. This might be related:

https://hazelcast.com/blog/locksupport-parknanos-under-the-hood-and-the-curious-case-of-parking/

This is going to be tricky to resolve as long as parknonos is used.

 

If there is any debug data we can provide you, let us know.

 

migrated

On FreeBSD 11.2, CPU usage in 1.14.4-pre6 is now back to the same sort of numbers I was seeing in 1.13.2 - it's slightly higher in parts (e.g. 1-2% versus 0.5-0.7% when empty, or 15-25% versus 15-20% with a single player in the Overworld), but that can probably be attributed to new code in 1.14 that hasn't been fully optimized, and it's far better than the 60-70% I was getting before.

For reference, the machine in question is running in VirtualBox 5.2.30 with 2 cores of a 3.6 GHz Core i7 4790 CPU.

migrated

Tested on Ubuntu Server 18.04.2 LTS - same type of reduction ratio as Ubuntu 16 (maybe a bit more)
From 16% to 10% on a X5670 CPU (1.14.3 --> 1.14.4 Pre-Release 6)

Not near as low as 1.13.2, which idles at <1% but none the less a noticeable improvement.

I appreciate the effort made thus far, but I still think the root cause of the high idle CPU has not been resolved and a lot more can be saved.

 

 

 

migrated

Seeing higher idle CPU on release 1.14.4.

Upgraded to release 1.14.4 from 1.14.3 where idle was at approximately 20% CPU. Idle is now 67%.

This is running on an Intel NUC D34010WK i3 processor, using docker on Libreelec/Kodi. Kodi itself is running at 4%.

migrated

Affects 1.14.4. CPU pegged at 100%.

Linode instance
2 CPUs
4 GB RAM
Debian 9

migrated

I tried the timerslack tweaking workaround suggested in the earlier comments. Sadly, I saw barely any improvement: ~17 % with default timerslack vs ~15 % with 200000–1000000 ns, all on an Intel(R) Celeron(R) CPU G1610T @ 2.30GHz.

As part of my tests I wrote a small C program that applies a custom timerslack value to programs started through it. This may be useful for anyone who is seeing notable improvements with a tweaked timerslack value. The program requires neither root privileges nor manual intervention on server restarts.

#include <sys/prctl.h>
#include <error.h>
#include <errno.h>
#include <unistd.h>

int main(int argc, char **argv) {
        if (argc < 2)
                error(1, 0, "usage: %s <command> [<args>]", argv[0]);

        if (prctl(PR_SET_TIMERSLACK, 1000000L, 0L, 0L, 0L) != 0)
                error(1, errno, "prctl");

        execvp(argv[1], argv + 1);
        error(1, errno, "execvp");
}

Usage instructions:

  1. Install GCC if it's not on your system already (e.g., on Debian: sudo apt install gcc)

  2. Put the code above in a file (e.g. copy, cat > slack.c, paste, Ctrl+D to finish writing)

  3. Optionally tweak the timerslack value in the prctl() call to whichever value works best for you

  4. Compile it (gcc slack.c -o slack)

  5. Run your server through it by prepending the program to the command that starts your server:

    ./slack java -Xmx1G -Xms1G -jar server.jar
migrated

The Paper project (fork of Spigot) has a pull request open to fix this problem:

https://github.com/PaperMC/Paper/pull/2342

Sadly, I haven't tried it since I no longer have access to the server that wasn't working properly.

 

migrated

Still an issue as of 1.14.4.  I'm running on a Linode server with 1 CPU, 2GB RAM.  Recently updated all the way from 1.10.2 (through each subsequent version) up to 1.14.4.  Can anyone spot when I did this?  sarcasm

[media]

 

It runs TERRIBLY under 1.14.4.  Are we serious?  Mojang added some extra animals and a few crafting recipes and our CPU usage goes through the roof?  It went from 5% CPU to over 70% CPU... and they call this issue resolved?  Please fix this.

Can anyone else suggest anything?  I'm really at a loss here.

violine1101

@unknown: Please create a new ticket and include as many details as possible.

migrated

So I have 3 servers running for my kids which are barely used. 
My minecraft server now acts more like a electrical heat source, not a huge problem now winter is coming.

Anyway, I thought about just putting the java process into sleep mode when there are no users connected and waking it up when a user tries to connect. It could actually be used for other server client applications as well.

The method works but it's rather dirty (not quick, because it took me around 20 hours to find out 😉)

I made a script that checks netstat for ESTABLISHED connections to 25565. If there aren't any, put all java servers into STOP mode. This way the processes are just put into sleep and they will not take any CPU anymore. They can also quickly be CONTinued if necessary. 
Below the contents of the script. You can put it somewhere in your home folder and call it minecraft_sleep.sh

#!/bin/sh
# Check if there are still connections made to minecraft server, otherwise PAUSE the process so that it won't take any CPU anymore.
# Also check /lib/systemd/system/knockd.service
if ! netstat -tn | grep 25565 | grep ESTABLISHED ; then killall -u minecraft -STOP java ; fi

 

Don't forget to make it executable:

sudo chmod ugo+x minecraft_sleep.sh

 

Then make a cronjob that runs that script every few minutes. Edit cronjob:

sudo crontab -e

 

Put this in the cronjob:

*/10 * * * * /home/michiel/minecraft_sleep.sh

Ok, sleeping the process is finished, now let's wake it up when a user connects.

I installed knockd which is a port knocking application that let you run a command when specific ports are knocked. 

sudo apt-get install knockd

 

I figured that knocking on the default 25565 should be sufficient to start the process again so I changed the configuration file at /etc/knockd.conf:

[options]
# UseSyslog
 logfile = /var/log/knockd.log
[openMineCraft]
 sequence = 25565
 seq_timeout = 1
 start_command = /usr/bin/killall -CONT java
 tcpflags = ack

 Edit: Isaac (see a few posts below) found out that he had to include a user for the start command (-u USERNAME). Since my start command ran as root it was probably not needed.

The above configuration has only one sequence, port 25565. Then, when a client connects on that port of the server, knockd invokes '/usr/bin/killall -CONT java' which CONTinues all java process which were STOPped before by the cronjob.

One problem of the knockd service is that it was not able to execute the command "sudo /usr/bin/killall -CONT java" which took me hours and hours to find out. 
In the end I found out that there is a service file for knockd located at '/lib/systemd/system/knockd.service'
You have to add some extra permissions for knockd to be able to CONTinue the java processes. Make sure CapabilityBoundingSet= looks like below, and don't put any linebrakes in between (credits to Isaac too). Copy paste of this code below doesn't seem to work because of those linebrakes.

CapabilityBoundingSet=CAP_NET_RAW CAP_NET_ADMIN CAP_CHOWN CAP_DAC_OVERRIDE CAP_DAC_READ_SEARCH CAP_FOWNER CAP_FSETID CAP_KILL CAP_SETGID CAP_SETUID CAP_SETPCAP CAP_LINUX_IMMUTABLE CAP_NET_BIND_SERVICE CAP_NET_BROADCAST CAP_NET_ADMIN
 CAP_NET_RAW CAP_IPC_LOCK CAP_IPC_OWNER CAP_SYS_MODULE CAP_SYS_RAWIO CAP_SYS_CHROOT CAP_SYS_PTRACE CAP_SYS_PACCT CAP_SYS_ADMIN CAP_SYS_BOOT CAP_SYS_NICE CAP_SYS_RESOURCE CAP_SYS_TIME CAP_SYS_TTY_CONFIG CAP_MKNOD CAP_LEASE CAP_AUDIT
_WRITE CAP_AUDIT_CONTROL CAP_SETFCAP CAP_MAC_OVERRIDE CAP_MAC_ADMIN CAP_SYSLOG CAP_WAKE_ALARM CAP_BLOCK_SUSPEND CAP_AUDIT_READ

These permissions are too many but I'm too stupid to find out which are necessary and which are not. I'm already happy that it's working 😃

After changing the service file, reload systemctl and the knockd service:

sudo systemctl daemon-reload
sudo systemctl restart knockd

It took me many hours and it's working finally. I thought let's share it with the rest of the world.

Cheers,
Michiel

 

 

migrated

Thanks Michiel!

migrated

Is anyone else seeing this again on 1.15.2? Running on a Nanode on Linode.com, just got my first over 90% for 2 hours notification. Top reports ~7%

migrated

Yes, it has never been below 25% CPU usage on an empty server without players.

But don't worry, this issue is eternally resolved, so have a nice day and don't worry about your CPU usage.

migrated

I see this on a local VM server.  The host VM server is an 8C/16T Xeon Debian Buster system running KVM, the guest Minecraft VM is also Debian Buster (4CPU/2GB RAM) using the VirtIO drivers for both NIC and disk. 

Inside the guest VM, the java process sustains somewhere in the 13% - 20% CPU, with occasional spikes higher.  Essentially no disk wait, nor do I see much system time.

On the host VM server, the qemu-system-x86_64 process consumes around one full core – 100% CPU, plus or minus.  Which explains @Chandler's Linode alerts.

In both of these cases, 100% CPU = 1 core/thread.  200% CPU = 2 cores/threads.

To simply look at top and other monitoring tools, you'd be at a loss as to why it looks so much higher on the host than inside the guest.  Disk activity is a few kilobytes every few seconds, network activity is also low, mostly the ssh traffic to monitor the guest.  Based on one of the duplicates (MC-154629), it looks like the server is ... well, it's not technically spin-locking, but may as well be.

Red Hat apparently found a solution for it: https://access.redhat.com/solutions/22965  Wish I knew what it was.

The sleep/knockd trick @michiel suggested sadly doesn't work.  Upon waking up, the server thinks a tick took the entire time the server was asleep and assumes the Minecraft server process just crashed.

FaRo1

Qemu is a hardware emulator, it's always insanely slow. I'm surprised that you even attempted to run Minecraft in it, I was barely even able to boot a tiny operating system and open a few menus in it when I ran Qemu on a pretty powerful computer. And that took hours. It's definitely normal that Qemu uses 100% of at least one CPU core.

migrated

Do not worry. 
Nobody cares about this issue which is still present on 1.15.2 servers as well.
The bug is "resolved" but isn't in reality.

Reminds me of my corporate days when helpdesk people were paid for every ticket they were able to close.
Tickets were closed without solving the problem but they reached their target.

My 3 java servers are all still running 25% CPU on idle.

It sucks but CPU time is probably cheaper than the time needed to fix this issue.

 

 

 

FaRo1

What is your exact issue? Just performance in general? Bug reports are supposed to give specific details about what the exact problem is. Just bad performance can have tons of reasons, like a slow computer, other things running at the same time, a lot of entities being crammed into a space and so on. If you can point to a specific issue that you know the cause of, please create a new report.

migrated

My issue is that a server without any players consume around 25% CPU  (from probably one core).
This is a screenshot from my kodi mediacenter which also acts as minecraft server for my sons and their friends.
Servers are only accessed a few times a week but I want to have them available all the time.

The machine is not a very powerful machine, but all other (idle) processes use a lot less CPU.

Linux kodi 5.3.0-53-generic #47~18.04.1-Ubuntu SMP
memory 15GiB System memory
processor Intel(R) Celeron(R) CPU N3150 @ 1.60GHz

[media]
migrated

I agree there is still something going on that should be dealt with on idle.  1.13.2 ran at 1% use idle (see my 1.13.2 v 1.14.x graph above).  1.15.2 on my i5-2500k home server runs at 6.5% average on idle.  That scales with the 14% idle I saw on my Q6600 on 1.14.4, and it scales with Michiel's percentages on that CPU.  SOMETHING in the server is doing work that shouldn't need to be when the server is idle and no players are on; nothing is being updated or tracked.  The server should easily be able to settle to <1% at idle within a few minutes of no connections like EVERY other process on my server does (and I have several that can easily become CPU hogs).  The increase is substantial enough to consume more CPU power over time than 1.13.2 and previous ever did.  It doesn't seem like much, but a constant low load like that can cause a processor to bounce up in it's power draw. 

In the charts I posted when I was using my older Q6600 (MC server CPU test), you'll also notice that EVERYTHING scaled up with the increase in idle use on the server side.  Even starting the server now uses much more CPU than 1.13.2 did.  Simple tasks do as well.  All of this seems to point to a potential bug somewhere that may be calling a routine too often/unnecessarily, etc.  I just spun up a 1.13.2 server to check on the same i5-2500k CPU, and yup, it falls to 1% avg.  If I manually watch it, it sits at near 0% most of the time with an occasional bounce to 6.5%.  My guess/theory is that since 1.14, the thread is perhaps no longer putting itself to sleep all the time, or is checking/running a routine constantly without regard to it being in a no player state when it could put itself to sleep/stop doing The Thing.  Probably a flag somewhere not set or removed.

migrated

You guys have to realize that even without any players online the server will still load and tick the spawn chunks unless you changed settings/modified your server to avoid that. With the chunk handling changes that were made starting in 1.13 this process has indeed been more impactful but comparing an active game server to any other idle process that does nothing but wait is a bit unfair.

migrated

With no players on, the server does not load and tick the spawn chunks by default.  For what reason would the spawn chunks continue to tick with no players on unless you deliberately set it to?  E.G. I have an iron farm (and usually do) in my spawn chunks, and it certainly does not run when no players are online, nor do any other things in spawn.  There are certainly reasons on MP servers to force it to do so (e.g. using /forceload), but by default, entities don't process, redstone doesn't work, etc.  That's the point.  NOTHING should be or is active to be tracked/ticked.  Nothing needs to be.  There is no work that needs to be done with no players on.  The world is effectively frozen until a player comes online.  The server therefore shouldn't have anything to track, so there is nothing for it to do, ergo, it's wasting CPU cycles and power doing whatever it's doing.

Per the official Wiki:
Idle timeout

Each dimension has its own "idle timeout." Without a player or forceloaded chunks in the overworld, this timeout eventually expires. When the timeout expires, some behaviors such as entity processing stop for spawn chunks and the rest of the dimension. The timeout can be reset by frequently sending entities through a portal.

Once the spawn chunks hit idle timeout when all players leave, the world is idle.  Nothing is supposed to be happening.  So, why the extra CPU cycles?  I suggest that somewhere in the code the idle flag is possibly not setting the thread to allow it to sleep, some other worker thread is busy doing useless tasks because it's hasn't been told it's ok to stop, or there is a bug that's continually calling code more than necessary underlying everything.  I don't know, I don't have the code, but it's behavior that's unnecessary, and therefore a bug.

FaRo1

There are a bunch of issues with chunk loading in recent versions, maybe that also affects spawn chunks in 1.13, which got fixed in 1.14 or 1.15. Please compare to 1.12.2. Also, are you using the same world in both versions in your test, meaning a world that is normally in the old version, one copy upgraded, one not?

migrated

@michiel The instructions you gave for stopping and starting the minecraft process were very helpful, but there were a few minor issues I found that might save anyone else who wants to attempt this a bit of time. First of all, in the start-command you need to add the option -u [name of user] for it to work. Additionally, I had a bit of a headache trying to figure out why the permissions were not working before realising that I had copied and pasted the long list of permissions with a couple of line breaks inbetween, which obviously breaks everything. Just make sure you remove the line breaks, and everything should work. Thank you very much for these instructions michiel, they have helped me a lot!

migrated

@Isaac Great I was able to help you! It makes my day a bit better.
I added your findings to my post.
Have a nice day.

violine1101

In case it isn't clear: This ticket was closed as fixed. For any performance issues that are still in the game, please create a new ticket and include as much information as possible.

migrated

Can you please tell me what the fix was?
I have read this thread again but I can't find it.

migrated

thank you @Michiel - that looks like a very useful workaround.  like you, many hours to get things working right here too.  dadops.  i found a similar script here that uses netcat instead of knockd.  although they kill the process instead of pausing it (probably good if you want the ram back), but i know i'd get guff from my users if it took 30 seconds to restart, so `kill -CONT` sounds much  better, but i guess it needs some testing to see if any plugins/etc will complain.

consider adding your script to  https://minecraft.gamepedia.com/Tutorials/Server_startup_script  if you don't add it, i might do so.  i use something like their systemd script, which is great when you have more than one server running.

violine1101, while some performance issues may have been implemented, what is a "normal" CPU load for a minecraft server?  i don't see this described anywhere, so is what we are seeing is now the expected performance?  maybe that just is-how-it-is!   it would be nice to have some reassurance that there is a real need for this CPU load rather than being a regression, which it seems many people suspect (and i trust the papermc folks have looked hard at this) https://github.com/PaperMC/Paper/issues/2336

i see ~20-25% cpu  with no users.  running 3 instances of papermc, ubuntu 18.04, i3-5010U (2.10GHz, 2 core, 4 thread), 16gb ram.  some have no plugins, some have several, CPU no different.

is running papermc the issue?  would running vanilla mean less cpu?  (somehow i doubt it)

i agree with the comments on electricity use.  not a scratch on bitcoin mining perhaps, but wasted cpu cycles are carbon pollution and that matters on scale.  concerned folks using workarounds won't make much of a scratch, but we do it on principle.  would be better to have a fix in the code. 

as mentioned above, may be parkNanos as @momcilo mentioned.  the article he linked in particular (https://hazelcast.com/blog/locksupport-parknanos-under-the-hood-and-the-curious-case-of-parking/) and the fact that parkNanos is OS dependent, so maybe there are no performance regressions in windows or whatever is used in the developer QA tests.  update: i spent quite a bit of time looking here, and i'm not clear this is helpful.  an idle server uses ~20% cpu, and pretty much all in "Server thread" (`jstack $PID | grep cpu=`) eg, 

 "Server thread" #20 prio=5 os_prio=0 cpu=388166.97ms elapsed=1866.16s tid=0x00007f4860f5b800 nid=0x7392 runnable [0x00007f47f57e8000]

 

migrated

here is my take on minecraft-sleep.sh:  https://gist.github.com/carchrae/3dc0da023c8e9bb36441dcd439950870

upside of my script; no root access/knockd required and can be run as regular/restricted user.  this script also resumes the server every so often.  papermc will restart if it hangs up. 

the server log is pull of papermc warning about not being able to keep up (and giving stack traces of where it stops).  one thing that is clear is there is a lot of state management going on even with no users.  it isn't just parkNanos or something easy like that.  from fish flapping to pathfinding, there is a lot running on the server even without users.

overall, i think it would be much cleaner to shut the minecraft server off on inactivity and let the first person to connect endure the 30-60s startup time.  anyway, this is faster to connect and nothing bad has happened, yet...

 

> Can't keep up! Is the server overloaded? Running 7702ms or 154 ticks behind

 

FaRo1

Again: There is a LOT happening in a Minecraft world that is completely independent of players being online. I don't think anyone here has so far reported very high CPU usage with an actually new world with no machines, mob collections, etc. in it. Of course those things require CPU, because they do work!
Also, saying "25%" doesn't help much. Please specify if it's 25% of one core or of all summed up and what the maximum speed of the CPU is. And again, don't complain about lag if there are things in your world that cause lag!
This report was about constantly near 100% usage of one CPU core at all times, even in completely empty worlds. Play 1.14.0 if you want to see an example.
Paper is of course not valid on the bug tracker, because it's not made by Mojang. And yes, mod loaders often make performance quite a bit worse and mods especially.

migrated

Fry

Community Consensus

Important

Performance

Minecraft 1.14 Pre-Release 5, Minecraft 1.14, Minecraft 1.14.1 Pre-Release 1, Minecraft 1.14.1 Pre-Release 2, Minecraft 1.14.1, Minecraft 1.14.2 Pre-Release 2, Minecraft 1.14.2 Pre-Release 3, Minecraft 1.14.2, Minecraft 1.14.3, Minecraft 1.14.4 Pre-Release 6

Minecraft 1.14.4 Pre-Release 6

Retrieved