Wanted to host a Minecraft server in a VM for a friend.
I've noticed that the CPU usage on the hypervisor host is really high 100-200%, while the VM is showing pretty low CPU usage.
Running dstat showed that the VM is doing a lot of interrupts and context switching even when the server is completely empty.
Host's top when the Minecraft server is stopped:
[media]Dstat output after starting the server, server is empty and there's not much going on except for the Minecraft Java process on the VM:
[media]On the Hypervisor host, the CPU usage for qemu-kvm skyrocketed:
[media]Let's check thread info from ps:
root@minecraft-igor-vm-1:~# ps -eLl | grep 416
4 S 999 416 1 416 0 81 1 - 1201481 futex_ ? 00:00:00 java
1 S 999 416 1 485 0 81 1 - 1201481 futex_ ? 00:00:03 java
1 S 999 416 1 488 0 81 1 - 1201481 futex_ ? 00:00:01 GC Thread#0
1 S 999 416 1 489 0 81 1 - 1201481 futex_ ? 00:00:00 G1 Main Marker
1 S 999 416 1 490 0 81 1 - 1201481 futex_ ? 00:00:00 G1 Conc#0
1 S 999 416 1 501 0 81 1 - 1201481 futex_ ? 00:00:00 G1 Refine#0
1 S 999 416 1 504 0 81 1 - 1201481 futex_ ? 00:00:00 G1 Young RemSet
1 S 999 416 1 518 0 81 1 - 1201481 futex_ ? 00:00:00 VM Thread
1 S 999 416 1 520 0 81 1 - 1201481 futex_ ? 00:00:00 Reference Handl
1 S 999 416 1 521 0 81 1 - 1201481 futex_ ? 00:00:00 Finalizer
1 S 999 416 1 523 0 81 1 - 1201481 futex_ ? 00:00:00 Signal Dispatch
1 S 999 416 1 524 1 81 1 - 1201481 futex_ ? 00:00:15 C2 CompilerThre
1 S 999 416 1 525 0 81 1 - 1201481 futex_ ? 00:00:03 C1 CompilerThre
1 S 999 416 1 526 0 81 1 - 1201481 futex_ ? 00:00:00 Sweeper thread
1 S 999 416 1 527 0 81 1 - 1201481 futex_ ? 00:00:00 Service Thread
1 R 999 416 1 528 0 81 1 - 1201481 - ? 00:00:00 VM Periodic Tas
1 S 999 416 1 529 0 81 1 - 1201481 futex_ ? 00:00:00 Common-Cleaner
1 S 999 416 1 602 0 81 1 - 1201481 futex_ ? 00:00:01 GC Thread#1
1 S 999 416 1 603 2 81 1 - 1201481 futex_ ? 00:00:25 Server-Worker-1
1 S 999 416 1 604 0 81 1 - 1201481 futex_ ? 00:00:00 G1 Refine#1
1 S 999 416 1 605 0 81 1 - 1201481 futex_ ? 00:00:00 Snooper Timer
1 S 999 416 1 606 0 81 1 - 1201481 futex_ ? 00:00:00 Server Infinisl
1 S 999 416 1 607 6 81 1 - 1201481 futex_ ? 00:00:59 Server thread
1 S 999 416 1 609 0 81 1 - 1201481 futex_ ? 00:00:00 ObjectCleanerTh
1 S 999 416 1 610 0 81 1 - 1201481 ep_pol ? 00:00:00 Netty Server IO
1 S 999 416 1 628 0 81 1 - 1201481 futex_ ? 00:00:00 chunk IO worker
1 S 999 416 1 629 0 81 1 - 1201481 futex_ ? 00:00:00 poi IO worker
1 S 999 416 1 630 0 81 1 - 1201481 futex_ ? 00:00:00 chunk IO worker
1 S 999 416 1 631 0 81 1 - 1201481 futex_ ? 00:00:00 poi IO worker
1 S 999 416 1 632 0 81 1 - 1201481 futex_ ? 00:00:00 chunk IO worker
1 S 999 416 1 633 0 81 1 - 1201481 futex_ ? 00:00:00 poi IO worker
1 S 999 416 1 634 0 81 1 - 1201481 poll_s ? 00:00:00 RCON Listener #
1 S 999 416 1 635 0 81 1 - 1201481 futex_ ? 00:00:00 Server WatchdogLet's measure syscall usage on the system:
syscount-perf -d 3 -c
Tracing for 3 seconds...
SYSCALL COUNT
exit_group 1
nanosleep 1
wait4 1
close 2
epoll_wait 3
inotify_add_watch 4
newstat 5
poll 7
getrusage 10
ioctl 632
sched_yield 18531
futex 37244Let's double check if our minecraft server is to blame:
syscount-perf -c -p 416
Tracing PID 416...
SYSCALL COUNT
epoll_wait 3
poll 5
getrusage 10
sched_yield 14747
futex 29638Let's get a look into strace from 607:
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99535}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99533}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99491}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99442}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99500}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99484}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99474}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99545}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99520}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99291}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99496}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99542}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99450}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99507}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99489}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99460}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99454}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99442}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99500}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99449}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99453}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99488}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99481}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99520}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99461}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99577}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99527}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99512}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99462}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99449}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0
futex(0x7f8c08feabd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=99468}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f8c08feab88, FUTEX_WAKE_PRIVATE, 1) = 0
sched_yield() = 0Now let's try Minecraft Server 1.13.2:
CPU usage on host is low:
No excessive context switching or interrupts on guest:
[media]Threads:
4 S 999 841 1 841 0 81 1 - 1255053 futex_ ? 00:00:00 java
1 S 999 841 1 842 1 81 1 - 1255053 futex_ ? 00:00:03 java
1 S 999 841 1 843 0 81 1 - 1255053 futex_ ? 00:00:00 GC Thread#0
1 S 999 841 1 844 0 81 1 - 1255053 futex_ ? 00:00:00 G1 Main Marker
1 S 999 841 1 845 0 81 1 - 1255053 futex_ ? 00:00:00 G1 Conc#0
1 S 999 841 1 846 0 81 1 - 1255053 futex_ ? 00:00:00 G1 Refine#0
1 S 999 841 1 847 0 81 1 - 1255053 futex_ ? 00:00:00 G1 Young RemSet
1 S 999 841 1 848 0 81 1 - 1255053 futex_ ? 00:00:00 VM Thread
1 S 999 841 1 849 0 81 1 - 1255053 futex_ ? 00:00:00 Reference Handl
1 S 999 841 1 850 0 81 1 - 1255053 futex_ ? 00:00:00 Finalizer
1 S 999 841 1 851 0 81 1 - 1255053 futex_ ? 00:00:00 Signal Dispatch
1 S 999 841 1 852 12 81 1 - 1255053 futex_ ? 00:00:25 C2 CompilerThre
1 S 999 841 1 853 1 81 1 - 1255053 futex_ ? 00:00:02 C1 CompilerThre
1 S 999 841 1 854 0 81 1 - 1255053 futex_ ? 00:00:00 Sweeper thread
1 S 999 841 1 855 0 81 1 - 1255053 futex_ ? 00:00:00 Service Thread
1 S 999 841 1 856 0 81 1 - 1255053 futex_ ? 00:00:00 VM Periodic Tas
1 S 999 841 1 857 0 81 1 - 1255053 futex_ ? 00:00:00 Common-Cleaner
1 S 999 841 1 859 0 81 1 - 1255053 futex_ ? 00:00:00 GC Thread#1
1 S 999 841 1 860 0 81 1 - 1255053 futex_ ? 00:00:00 G1 Refine#1
1 S 999 841 1 2369 0 81 1 - 1255053 futex_ ? 00:00:00 Snooper Timer
1 S 999 841 1 2370 0 81 1 - 1255053 futex_ ? 00:00:00 Server Infinisl
1 S 999 841 1 2371 5 81 1 - 1255053 futex_ ? 00:00:09 Server thread
1 S 999 841 1 2373 0 81 1 - 1255053 futex_ ? 00:00:00 ObjectCleanerTh
1 S 999 841 1 2374 0 81 1 - 1255053 ep_pol ? 00:00:00 Netty Server IO
1 S 999 841 1 2375 0 81 1 - 1255053 futex_ ? 00:00:00 WorldGen-Schedu
1 S 999 841 1 2376 7 81 1 - 1255053 futex_ ? 00:00:13 WorldGen-Worker
1 S 999 841 1 2377 0 81 1 - 1255053 poll_s ? 00:00:00 RCON Listener #
1 S 999 841 1 2378 0 81 1 - 1255053 futex_ ? 00:00:00 Server Watchdog
1 S 999 841 1 2379 0 81 1 - 1255053 futex_ ? 00:00:01 File IO ThreadNow let's check syscall count:
syscount-perf -d 3 -c
Tracing for 3 seconds...
SYSCALL COUNT
exit_group 1
wait4 1
close 2
epoll_wait 3
poll 6
getrusage 10
ioctl 632
futex 5726This issue seems to be connected with: MC-149018
Maybe some multithreading changes or something messed up the performance for VMs?
It looks like it all started in 1.14.
Let me know if there's anything more I can add to the report to help you with resolving this.
Linked issues
Attachments
Comments 14
Hi!
Same problem here, running debian 10 64 bits on host and guest.
I think this is the same problem described at MC-154629 , incorrectly marked as "Solved"
Please fix this.
Thank you.
How about 1.16.1
Hi,
i can confirm this issue. It is not fixed in the latest version. To reproduce this you just need to make a linux VM with KVM, start a server inside it and the look at the cpu usage of the vm and of the host using "top"
I've been running a vanilla Minecraft server with no add-ons for many years, and I can confirm that this issue is still present in 1.16.2. Because so many Minecraft servers are hosted on virtual private servers, this issue seems fairly serious. I have tested this with a KVM hypervisor running Debian 10 and a guest running Debian 10 (CLI only) with 6GB of RAM dedicated to it. I tried:
Setting the vCPU allocation to 1, 2, and 3 in separate tests;
Setting the ethernet interface as Virtio or e1000 in separate tests;
Changing between ePoll and default channel types (use-native-transport in server.properties);
Changing various clock sources in the guest.
Note on that last bullet point that the host is using TSC as its clock source. The guest, as is expected, locks out TSC as unstable and uses kvm-clock (preferred), hpet, acpi-pm, etc. instead. Without native TSC, all other clock sources are virtualized which, I believe, involves context switching to the hypervisor for timing inquiries.
While I don't understand the structure of Minecraft server under the hood, I would expect that if it uses spin-waits that involve timer checks, it may result in a ton of context switching to the hypervisor to service those. This could possibly be overcome by estimating the typical time each cycle of the spin takes and spinning a certain number of times between clock checks to minimize the context switching while delivering a reasonably high degree of timing accuracy.
These images were taken from a completely idle 1.16.2 instance a few minutes after it completed its start-up.
[media]Information from "top" as seen from the guest running an idle Minecraft server. The java process is shown.
[media]
Information from "virt-top" as seen from the host. The subject guest is shown.
[media]
Information from "top" as seen from the host. The subject guest process is shown.
Note that this appears to be the same or related to the issue listed in MC-154629, which a bot claimed was resolved but appears to have not been.
To the developers: Thank you for your time looking into this issue.
I've just been hit by this on 1.16.3 too, and wanted to share my findings, including a (non-trivial) workaround.
They seem to confirm what others in this thread already suspected: That minecraft calls a combination of sched_yield and futex with an insanely low timeout way too often, thereby burning A LOT of CPU for absolutely nothing.
If you look at the strace-output in the original report, you can see it calls _futex(FUTEX_WAIT_PRIVATE)_ with a timeout of 100000 nanoseconds. That's 0.0001 seconds. Or put otherwise, it requests no less than 10000 schedule-away-then-wakeup-events every second.
That is a giant waste of CPU even on real hardware, as context switches are expensive, and it will certainly prevent any meaningful powersaving by the CPU. The reason it has such a much more devastating effect on VMs is probably that the overhead there is much higher, and it's one of the few things that the virtualization-hardware-acceleration of current CPUs cannot handle in hardware.
Somewhere else I found the info that just running FreeBSD instead of Linux in the VM makes Minecraft behave an order of magnitude less CPU-hungry, and I can confirm that finding: Running the exact same minecraft-version and map in a VM on the same host, with the only change being FreeBSD as the OS inside the VM instead of Linux, makes the CPU-usage on the host drop from 100% to 5% when the server is idle.
My attempt at an explanation for this was that FreeBSD still uses a fixed-tick-based timing system, defaulting to 100 HZ, so even though Minecraft requests 10000 wakeups per seconds, it will never get more than around 100. Linux on the other hand nowadays by default is tickless and supports high precision timers, meaning you can schedule things in arbitrary intervals there, even something idiotic like 10000 wakeups per second. You can however still change that default, and that is what I did:
I recompiled the Ubuntu-4.15-Kernel, with one mayor change compared to their default-config: instead of tickless-mode, 100 HZ ticks were forced (CONFIG_HZ_PERIODIC=y, CONFIG_HZ_100=y and CONFIG_HIGH_RES_TIMERS not set). The result of this should be that, just like on FreeBSD, Minecraft should get something like 100 wakeups per second even though it requests 10000.
The results speak for themselves: When this kernel is booted in a VM running two minecraft-servers, the CPU-usage of that VM on the host drops from 200% (two cores fully utilized) to 5% when the minecraft-servers are idle. That is a HUGE improvement. It also proves how much CPU Minecraft wastes for doing absolutely nothing if the kernel lets it.
Please developers, fix this insane waste of CPU cycles for idle servers (not only) in virtual machines. Considering how many minecraft-servers run in VMs somewhere, fixing this problem would probably save quite a few tons of totally useless carbon-dioxide-emissions world-wide.
I'm a kvm host provider, I have same problem!
Even 35% of all my 72 cores in cpu system computing.
@PoempelFox
Do you have a solution such as change some options in kvm xml?
Dear Pony233, this is a Bugtracker and not a discussion forum, please do not abuse it as such. If I had found a workaround that is simpler than "compile your own guest kernel", I would have written that. That does not mean that it is impossible that simpler workarounds exist, just that I have not found any.
Thanks to Michael Meier's additional information and workaround, I tried some related configuration changes. It doesn't solve the problem entirely, but using a stock Debian 4.19 kernel, I added the following two kernel configuration parameters in the Grub bootloader on the guest that runs Minecraft: nohz=off highres=off
I don't see a kernel parameter to drop the periodic tick to 100Hz from the Debian default of 250Hz.
However, just this change dropped the idle server from holding a core at 100% down to roughly 8% utilization. The game server seemed to run fine, although more testing is needed. It's still not a perfect fix, but this might be the easiest "fix" for a while.
This should work with other recent kernels and other Linux distributions, but I haven't tested on any other setups.
Thanks to Michael Meier and Robert C.
After modify grub cmd parameters "nohz=off highres=off", all is well!
Host: Centos7
Guest: Centos7
Can confirm that Robert C.'s simple workaround with using the kernel parameters "nohz=off highres=off" for the guest works just as well as my kernel recompiling on Ubuntu 18.04s 4.15 kernel. The effect of the fact that this does 250 Hz instead of 100 Hz seems to be absolutely neglegible on my system (<1% additional CPU utilization). Thanks, Robert C..
https://bugs.mojang.com/browse/MC-154629?focusedCommentId=958423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-958423
If you have access to your KVM host machine, you can run the following command as root:
echo 0 > /sys/module/kvm/parameters/halt_poll_ns
**I put it in my root crontab to run at startup.
Michael Meier seems to have a better understanding than I do, but I'll give a quick explanation of what I believe is happening host-side:
When a vCPU halts, KVM will continuously poll the vCPU for a short while after in case there's a wakeup right after. This is good for responsiveness of the guest, but means that, in layman's terms, KVM is essentially running around in a while() loop. If there isn't a wakeup in a certain amount of time, KVM will pass it off to the scheduler, meaning it polls the vCPU a lot less often.
Minecraft breaks this by having a ton of wakeups really close together. So, the vCPU sleeps, has a wakeup right after, Minecraft does a really teensy calculation, and then the vCPU halts again. Rinse and repeat. That means that the majority of the time, KVM is simply polling the vCPU for wakeups because Minecraft thinks it needs to wakeup really frequently for some reason, even when there's nobody online.
Since it's KVM running in circles, the host sees the high CPU. The guest doesn't see it, because as far as it knows, it spent most of the time with the vCPU halted. The command I pasted above disables this feature and forces KVM to only uses scheduling.
Now, as I said, this is what I think is happening based on the research and experimenting that I've done. There's probably some people who're more familiar with the Kernel and KVM on the development side that could better explain, and/or confirm/deny what I'm saying.
Why it's a concern for me, personally:
My server is hooked up to a UPS. With the vCPU doing this continually, it halves the time that the UPS can last.
Also, I have a Threadripper CPU, so I now have enough cores that I can afford to "waste", but that wasn't always the case. A few months ago, for several years prior, I was running KVM on a quad core CPU. I believe that the majority of people, especially as hobbyists, would be in a similar boat.
So, I would suggest that Mojang keep looking into this. A lot of people host their Minecraft servers on a VPS, so this could potentially impact them.
Hi,
I have some additional information that may be helpful, as well as a hack that can be used without recompiling the kernel or tweaking boot parameters.
I'm experiencing the same high CPU while idling problem on a 1.19 based modpack. The server is running directly on the host, so no VM involved in this case.
As Michael Meier described, I also observe a high number of syscalls, mainly:
clock_gettime
futex (with a timeout 1 microsecond from the time returned by clock_gettime)
futex (apparently to signal/wake up something else after the first futex)
Using the spark profiler in Minecraft, it seems like it's trying to sleep (using parkNanos) while waiting for the next event, but tracing the syscalls reveals that it appears to be using a very low timeout. This probably doesn't hurt much on Windows, but Linux has higher precision timers by default and will return more quickly. What looks like a reasonable wait on Windows effectively turns into a busy loop / spin on Linux with a tiny sleep that hides the problem slightly: I observe about 35% CPU (of a core) on the most offending thread instead of the 100% that would have clearly labelled this as a busy loop.
The problem here is not Linux, it's just that Windows (and probably also FreeBSD) doesn't expose a timeout that appears to be set way too low.
I came across this blog post that examines parkNanos and provided a simple way to tweak the Java process (and threads):
https://hazelcast.com/blog/locksupport-parknanos-under-the-hood-and-the-curious-case-of-parking/
Checking on my host, I have the same value of 50000 (50 microseconds) in /proc/pid/timerslack_ns. Using jstack to find the Java threads waiting on parkNanos, and setting timerslack_ns to a higher value for those threads significantly reduced the CPU load on my computer.
Since there are many threads, I made a quick hack in Python to find the Java threads belonging to the Minecraft server and set their timerslack to 5ms:
https://gist.github.com/jmbjorndalen/23e6922b8ef069014463ffc97d717506
With no users on, I went from around 50% CPU usage in total (35% from the main thread, the rest from a few other threads) to around 4-7% CPU usage on the host.
If you are curious, there is also a blog post that examines parkNanos on Windows:
https://dzone.com/articles/locksupportparknanos-under-the-hood-and-the-curiou
slicedlime
This is expected with the new chunk system. The server no longer sleeps, and instead spinwaits.
If you want to learn more, jump on the PaperMC Discord.