Running the 1.14.2 jar in a Debian 9.9 or 10 guest uses 100% of a core on the host when idle, with no players. With players, it uses at least 100% of a core. A `top' on the guest indicates the running server is using 6 to 8 percent of the vCPU.
/proc/interrupts shows the local timer interrupts increasing at around 180K/sec on the guest and host.
To reproduce:
1) On a Debian 9.9 system, craft a qemu-kvm guest with sufficient memory and disk
2) Download the 1.14.2 jar, start it with the flags given on the download page
3) Wait for the world to be created
4) Enjoy the "can't keep up" messages.
Profiling with visualvm seems to indicate a large majority of the time is spent in netty/epoll-wait. Really, like, almost all...
An strace, on the guest, of the most active thread just pours out the following:
futex(0x7f5be0a2d468, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=733}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f5be0a2d418, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f5be0a2d468, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=724}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f5be0a2d418, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f5be0a2d468, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=753}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f5be0a2d418, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f5be0a2d468, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=742}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f5be0a2d418, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f5be0a2d468, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=669}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f5be0a2d418, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f5be0a2d468, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=710}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f5be0a2d418, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f5be0a2d468, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=743}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f5be0a2d418, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f5be0a2d468, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=700}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f5be0a2d418, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f5be0a2d468, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=736}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f5be0a2d418, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f5be0a2d468, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=729}) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f5be0a2d418, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f5be0a2d468, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=710}) = -1 ETIMEDOUT (Connection timed out)
Seems a rather short time to wait...
On the host side, the qemu-kvm profiling seems to indicate that the guest thread is spending all of it's time servicing timer interrupts.
Environment
Host:
Debian 9.9 (stretch), vanilla with latest updates
Linux tochtli 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64 GNU/Linux
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Xeon(R) CPU E3-1246 v3 @ 3.50GHz
stepping : 3
microcode : 0x25
cpu MHz : 3799.926
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single ssbd ibrs ibpb stibp kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts flush_l1d
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips : 7000.16
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:
Guest:
<domain type='kvm'>
<name>ontest</name>
<uuid>cb423de8-b80d-43b1-b92d-fb7da6f16cc1</uuid>
<title>Lee Test</title>
<memory unit='KiB'>10485760</memory>
<currentMemory unit='KiB'>10485760</currentMemory>
<vcpu placement='static' current='1'>2</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-2.8'>hvm</type>
</os>
<features>
<acpi/>
<apic/>
<vmport state='off'/>
</features>
<cpu mode='host-model'>
<model fallback='allow'/>
<topology sockets='1' cores='2' threads='1'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/u1/vm-storage-pools/default/ontest.qcow2'/>
<target dev='vda' bus='virtio'/>
<boot order='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='hda' bus='ide'/>
<readonly/>
<boot order='2'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:6e:d6:53'/>
<source bridge='on0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'/>
<address type='virtio-serial' controller='0' bus='0' port='2'/>
</channel>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='spice' autoport='yes'>
<listen type='address'/>
<image compression='off'/>
</graphics>
<sound model='ich6'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</sound>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='2'/>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='3'/>
</redirdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</memballoon>
</devices>
</domain>
Java
openjdk 11.0.3 2019-04-16
OpenJDK Runtime Environment (build 11.0.3+1-Debian-1)
OpenJDK 64-Bit Server VM (build 11.0.3+1-Debian-1, mixed mode, sharing)
....or
java 11.0.3 2019-04-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.3+12-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.3+12-LTS, mixed mode)
Linked issues
Comments 14
No. It is not fixed, nor mitigated at all. Honestly, I just got tired of going around and around with the devs on the issue. If I'm remembering correctly, the final substantial activity from them was when they tried to reproduce and couldn't. I suspect they tried to reproduce using a Linux guest on a Windows host, which will not reproduce the issue.
Really sorry. I just gave up. The overhead I incurred to test plus the lack of meaningful effort on the part of the Mojang developers just left me feeling super frustrated.
As an aside, my efforts lead me to believe either the server is not using the Java Netty package in a nice way or that there is a bug in the Netty package. I checked the Netty bug reports and, yes, others are having the same problems with it. Still... See https://github.com/netty/netty/issues/5896 In that case, the reporting individual ditched it and wrote their own. Doesn't bode well for a fix in Minecraft then 😞
First of all, sorry for not seeing your message sooner - I still got to figure out how I make this Jira instance send me emails.
Thank you so much for responding! It's unfortunate that this seems to have not been looked at properly, maybe I'll try to get another bug report raised when I have some time for it. You're probably right that this is somehow linked to the netty issue (thanks for sharing that!) so if the MC developers do not recognize this on their side maybe we'll have to resort to using some external fix if there ever is one (not sure if this is even possible, I am not familiar enough with these codebases).
Anyway, at least it's good to know that this issue is not due to me doing something blatantly wrong. However, I am wondering whether it might be possible to get one of the alternative Minecraft server projects (Spigot / Paper) to fix this issue? I did experience the same issue with every alternative I tried, but maybe they'll be more responsive - I'll have a look into that.
Again, thank you for sharing your knowledge with me. Should I figure something out I'll make sure to share it here.
Till
Just signed up to comment I'm running into this as well. Debian 10 as KVM host and Ubuntu 18.04 as VM. Running 1.15.2 version of Minecraft with default-jre jvm.
Also experiencing the issue, Ubuntu 18.04 on both host and guest, hypervisor is KVM. MC 1.15.2 running on openjdk-8-jre-headless.
This is marked as solved, but the problem is not solved.
The problem is high CPU usage on host side, while low on guest side.
Switching guest OS to FreeBSD is not a solution neither.
Thank you for your report!
We're tracking this issue in MC-149018, so this ticket is being resolved and linked as a duplicate.
That ticket has already been resolved as Fixed. The fix will arrive in the next version or is already included in the latest development version of the game, you can check the Fix Version/s field in that ticket to learn more.
If you haven't already, you might like to make use of the search feature to see if the issue has already been mentioned.
Quick Links:
📓 Issue Guidelines – 💬 Community Support – 📧 Customer Support – ✍️ Feedback and Suggestions – 📖 Game Wiki
-- I am a bot. This action was performed automagically! Please report any issues in Discord or Reddit
This is still an issue as of 1.16.2. It also seems to have migrated to MC-183518 with additional information. It doesn't appear to be getting traction there either. I'm surprised considering how many MC servers are hosted on VPS. At the end of the day, it's just under $20 USD per year in wasted electricity for me, and having one fewer core to borrow idle time from in my host. In the meantime, I posted information in the comments on the above-linked issue.
I have found a workaround for this. After discussion on the VFIO discord
https://discord.com/channels/244187921228234762/244190447147286529/790969366476095501
they suggested to change a kernel parameter on the host `kvm.halt_poll_ns=0`
You can apply it on the host at runtime as root with
```
echo 0 > /sys/module/kvm/parameters/halt_poll_ns
```
The default value on archlinux seems to be 200000.
@Lee Ward
I am aware that I am resurrecting a very old issue here, but has this problem every been fully fixed for you? I am running into it with all Minecraft server versions >= 1.14 (KVM VM running Debian 9 / 10; host running Debian 9 / Ubuntu 19.04) and none of the reports marked as "resolved" here have offered a solution to this issue for me. I am kinda lost and very much hope that there has been some development concering this issue?
Thank you very much in advance,
Till