[MC-245817] Server constantly writes to disk at about 500kB/s

Since the release of version 1.18, the minecraft server constantly writes about 500kB/s to the disk, even if no players are currently online. I attached an iotop screenshot showing three IO-Worker threads of the java server executable, and a list of all currently open file descriptors of that process.

To me, this is an issue, because the server runs on an SSD drive, that wears down by write accesses.

Attachments

fileaccess-trace.lst

minecraft-server-1640040127143.apps

minecraft-server-lsof.lst

Comments 14

wobst.michael 2021-12-20T12:07:00Z

Cannot confirm. At least not with a freshly started, idling server. As expected, disk writes are only happening for me during client processing. Tested with both sync-chunk-writes set to false and true.

Bernhard Gräuler 2021-12-20T22:47:11Z

I investigated this a little further. Using fatrace i was able to find the file the server constantly accesses for write, it's a world region file (/usr/local/games/minecraft/world/region/r.0.0.mca)

In addition to that I created a VisualVM Application snapshot with some thread dumps and profiler snapshots, and attached it here. Hopefully this helps identifying the issue...

wobst.michael 2021-12-21T00:34:04Z

Could you please measure the total disk write for like 60 minutes using iotop -ao and see what it gives? Make sure that no user can enter the server during that time.

Bernhard Gräuler 2021-12-21T07:33:56Z

Attached some iotop screenshots. First one created shortly after I started the server, and for the last one, created around 60 minutes after startup, I messed up switching from thread to process view, expecting iotop to summarize the values. Yet, it actually resets the counters, so the values in the last screenshot must be added to the previous one, and the count between 30m and my mess-up is lost.

I also tailed the server log printing the connected players from time to time. This is a server I use to play the game with my kids, so as they are in school right now...

Charles Fabbri 2021-12-21T17:22:44Z

I'm also experiencing the same issues, see iotop pictures below. I'm running in a KVM virtual machine. Details below. This happened after upgrading to 1.18.1 from 1.17.1. Let me know if you need any more information.

VM Uname: Linux 5.13.0-22-generic #22-Ubuntu SMP Fri Nov 5 13:21:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
VM OS: Ubuntu 21.10 Server
Hypervisor: KVM 1:3.1+dfsg-8+deb10u8, running on Debian 10

CPU: Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz (2 cores allocated to VM)
Memory: 16GB total, 6GB allocated to VM

java -version: openjdk version "18-ea" 2022-03-15
OpenJDK Runtime Environment (build 18-ea+15-Ubuntu-4)
OpenJDK 64-Bit Server VM (build 18-ea+15-Ubuntu-4, mixed mode, sharing)

Server start command: /usr/bin/java -Xmx4G -jar minecraft_server.jar nogui

iotop

[media]

iotop -ao for 60 minutes.

[media]

4 more comments

Charles Fabbri 2021-12-22T03:26:33Z

Doing an fatrace, I'm getting the same file r.0.0.mca, I'm also getting writes to r.0.-1.mca as well.

[media]

wobst.michael 2021-12-24T16:29:37Z

Please have a look at MC-245910. Does anyone of you have a permanent running hopper contraption on the server?

Russell Piper 2021-12-24T19:47:57Z

I had opened MC-245906 for something similar. Post upgrade to 1.18 or 1.18.1 from 1.17.4 has created a never-ending update loop for one region file for me r.-1.0.mca.

I also tried doing a fresh 1.18.1 server and like Mike, I did not get constant hard disk writing.

Now, I do have campfires but I'm not burning anything on them (a village had generated almost at 0,0), but I do have one hopper clock running on an iron farm, but it isn't in that particular region file. I do have another hopper clock on a kelp farm that is in that file, but I turn that off/freeze it before I log off to try to prevent my flying machine from getting stuck.

My storage/sorting system is within that region file, but that isn't running a hopper clock anywhere. Hoppers are only unlocked when items go over the appropriate hopper.

Russell Piper 2021-12-24T20:24:35Z

I FOUND the problem on my world.

I have a mob farm that cycles a bunch of redstone repeaters as a timer with a single redstone torch that goes forever. I removed the torch and logged out and the writes stopped.

This wasn't a problem with 1.17.4 - I've had the same set up for years. So for the time being, I guess I'll add an on/off switch to that farm so it won't murder my SSD.

The hopper clock thing got me thinking, so I hopped on my world put out all of my campfires, checked, no changes, turned off all of my hopper clocks, checked, no changes, removed a rolling hopper minecart that goes forever, no changes. And then I remembered my mob farm, disabled that, and the writes have stopped.

Bernhard Gräuler 2022-01-03T07:34:20Z

I also can confirm this must be related to redstone-repeater clocks and / or hoppers. Can't say for sure, what exactly causes it, yet, but as soon as I deactivated those circuits the writing stopped.

Fred 2022-01-11T01:28:26Z

Hello, I just want to add that this is not specific to Linux, there is always a lot of things going on on my Windows server, and on a test-version in 1.18.1 I see constant writes of 3 to 4 MB p/second. I also see a loss of 5-10 mspt when I simply put a stack of items in a chest, while the items flow through the connected hopperline the mspt increase is 4-5 mspt, but mspt goes up to 10 extra mspt once they start being processed by a small redstone contraption that filters these items into stackable/unstackable, shulkerbox yes/no, empty/filled shulkerbox, and finally a shulkerbox-filler for stackable and non-stackable items. This is accompanied by an increase of amount of diskwrites.

I haven't done the exact math, but I am quite sure this behavior will trash my SSD within months.

Please look into this, if you need any more info let me know 🙂

Fred 2022-01-22T10:26:43Z

Hello, I did some more testing, ran my server as vanilla (normally I use Fabric + a bunch of optimizing mods), same behaviour except higher mstp drop when the stack is being processed (+14 mspt when being pushed through the hopperline and sorting, +6 mspt when only sorting) . I changed sync-chunk-writes from true to false in server.properties, that stopped the contineous saving to disk!!! But it did not change the mspt drop, so I guess those issues are not related.

Adrian Östergård 2022-02-03T09:36:38Z

Is this still an issue in 22w05a or later?

Fred 2022-02-03T12:44:01Z

I've upgraded a copy of my server to 22w05a with the forceupgrade and erasecache options as I did before, and ran it vanilla. I could not reproduce the excessive writing to disk nor the raised mspt with the same steps I used before, so I can confirm it is solved.

To be sure the sync-chunk-writes setting didn't influence the result I tested both with this setting on false and true. With both values this issue seems resolved.

Thank you very much for fixing this 🙂