Mineshaft.dat uses too much CPU

Ah yes, Mojang has "abandoned you" because we haven't dropped everything to work on this one specific bug out of thousands...

Other files: 6kt. Mineshaft.dat: 9mt. 😃 Thats why it lags when looking direction of mineshaft!

The 1.7 versions of Minecraft have resulted in the most baby sitting I've ever encountered since 1.8 Beta, before I realized I had to delete the Mineshaft.dat file with every restart. After it got to 11MB, it grows to 3MB in less than 24 hours. Not even the worst snapshot needed this much attention.

Before I determined the issue, I tuned the garbage collector, installed several watchdog scripts, did hourly restarts, did metrics based restarts. I troubleshot everything I could think of. Then, one of my regular players suggested deleting a random file. I couldn't believe this was the solution. But it was. The difference is night and day.

I believe any active world over 8GB will be crippled unless they have the resources to just overpower the problem. All I know is, I went from having to restart 15 times a day, down to only one or two restarts required.

MC-33134 appears related, and I mentioned it has a stack trace. My gut feeling, looking at the stack trace is that there is some kind of race condition. MC-34947, which is currently ranked as #47 in popularity, is also related, but the comments are all over the place. Obviously, the netty timeout is attracting every kind of connectivity failure, not just ones related to this. But I believe the most common bug-related reason people see a client timeout is related to this .dat growth problem.

Ok, hyperbole has no place in a publicly facing bug tracking system with thousands of reports. But I think this might be a near-universal problem.

Please don't be mad at us, Mog 😞

It's just that after finally figuring out the cause of the issue, we tried to get Mojangs attention to it, since we were sure that it a) is the root cause of the problem and b) could be the main cause for at least 100 other bug reports where ppl complain about serious lag and/or high ram/cpu usage and - most importantly - Read Timeouts (wich ppl still falsely blame netty for). Seeing that Mindcrack has the same lag-spike issues, I even tried to contact some of the Mindcrackers, since they seem to have a more direct link to Mojang, but as a single person that task seems even more difficult than contacting one of the Mojangsters directly...

So after trying to get attention to this report for weeks and beeing sure that it is a severe, universial problem, still seeing it as 'unconfirmed' is a frustrating feeling...

Though, nobody can blame you for that either, after all, reaching Mojang and having an influence on the development on Minecraft (in certain borders) is possible and happening all the time. I think this is something that makes Minecraft very unique.
It's just that in cases like this where there are hundreds of bug reports showing the effects but only one or two actually showing the cause of the problem (or at least giving a very good hint on where to look) the ods are ppl upvote the reports they find first - wich are the ones showing the effects of the bug and so you can easily feel left alone...

So please don't take it personally, we love Minecraft and we love Mojang, if it wasn't that way we wouldn't even commit to this bugtracker 🙂

Don't worry, I'm not upset, it's just that while this issue with mineshaft.dat may be causing these other issues, I still need to track down exactly why the file is becoming bloated in the first place. Unless there are comments that I'm not seeing, that question is still very much up in the air and will be a bit of a bear to track down.

Well, thanks for looking at it.

I've been having the same issue and my Mineshaft.dat file is about 100k. I've been noticing the process has been using up all of the RAM allotted to it on the command line and basically appearing to GC in a nice loop every 15 seconds. This was definitely not an issue in 1.6.x.

Though I understand server needs grow as more stuff is added over time, e.g. taller worlds, it seems like people with hardware an order of magnitude greater than mine are also seeing the issue.

I have a 32-bit dual core atom, 1.6ghz, 1.5GB max memory. I used the DEFAULT world type for this server, fresh for 1.7.x. Just restarting the server clears up the issue for a while, until enough stuff gets loaded by the server, e.g. entities because someone goes near my animal farm, then it's game over. I've run two or three instances on the same box in the past with no issue. And I can do so now, except after a while it goes into GC looping nightmare mode.

8.2% of 4.5GB (w/ swap) when I started, slowly climbing up to 8.7% now after 10 min, CPU is 47/400%. Eventually that will go up to 33% mem at which point it will GC, but be at 47% CPU while it's not GCing, which I have suggested to people is a good time to work with lava. 🙂

[in a flyish voice] "Help me!!!"

@Jim

You have a 32bit dual core Atom, I think that's your issue. :/

@Jim Well that would explain why I have better performance on my server with GC turned off, and why my server spikes to full proc usage (2.3MB mineshaft.dat) when I turn GC back on.

@TonyCubed Don't be so sure. I run my MC server off a core 2 duo and it outperforms some i3 due to the faster individual cores. All depends on how the server is configured (OS), how much overhead, etc.

@Talven81

A Core 2 Duo/i3 would smash the pants off an Atom, even a Celeron based CPU has a good chance depending on which version you get.

I've recently experienced high CPU and RAM usage in one of my worlds leading to stop and go mobs/player behavior due to this. This was in 1.6.4 vanilla.

http://www.minecraftforum.net/topic/2210743-solved-164-playermob-movement-lag/

I fixed it by narrowing it down to Mineshafts.dat through process of elimination, and deleting said file fixed my issues. That's when I Googled the file name and found this bug report on it. I guess this will continue to grow and need constant deletion from what other comments said, but it's at least made said world playable again.

Still having this issue, but deleting the Mineshaft.dat isn't helping as much as it used to because other .dat files are now getting too large as well. Specifically, the Fortress.dat file. It's now 7.5MB. Before, when the Mineshaft.dat file was the issue, it was 11MB. Deleting it was an acceptable workaround because it currently does nothing (that I'm aware of).

My server's total map size is 43GB. I delete the Mineshaft.dat every hour or so, when it gets to about 1MB. But deleting Fortress.dat is completely out of the question. I'd rather restart every 15 minutes.

This is such a known issue, modded server implementations have solutions, like disabling structures:

http://www.spigotmc.org/threads/disabling-structure-saving.8555/

... though this seems like a pretty lousy workaround. In order to find wither skeletons, one must generate new map terrain. Every time these structures are found, once they are unloaded from memory, players have to start over and find new structures. Screw that.

Confirmed on 1.7.4

Having High CPU usage and disconnects when exploring new terrain (Vanilla 1.7.4 created map on 1.7.2) Exploring new terrain is impossible now. Standing in the same spot for 15 minutes is not a problem, server load goes down. When i move a few blocks into unexplorered territory: server load goes to 100% and almost always disconnects me.

@Anthony While deleting .dat files works as a temporary workaround I still don't recommend it. While the mineshaft.dat is not directly used at this time, it could easily be used in the upcoming API for mineshaft specific mods and I would hate to see someone destroy functionality for the majority of their map (though I think I saw a rebuild option for that in MCEditor). Similar to what you said about deleting the fortress.dat... disabling structures would have the same issue.

@Talven81, I agree with you, but if the choice is destroy structure data or restart literally every single minute ... I do not exaggerate. The players on my server decided to travel by nether roof to 300,000 blocks. Vanilla has no way of preventing this kind of thing.

I hate deleting these files. I saw a comment on MC-33086 that claims this is an easy fix. I sure hope so, but the damage to my world has been done. Even if MCEdit can rebuild structures, I'm not sure if that's practical for a world over a certain size.

Hmm while no specifics were given the latest snapshot mentions "Overworld, Nether and End are stored differently, increasing performance" will be interesting to see if this effects this ticket.

@Anthony - Yeah I understand what you are running up against on your server, you don't have many options. As far as preventing players from going too far you can set up a command block to testfor any player outside of a certain range and TP them back to a safe spot. This would essentially create a TP wall not allowing them to go outside a specific range. Again not a fix for this problem but just an idea for you.

Very good suggestion. Basically a snowglobe. But prior to 1.6.4, it was never necessary to confine anyone, except to save hard drive space. And it becomes more complicated when we consider Nether travel. Nonetheless, it is nice that there is an option for that if it comes down to it.

Regarding the performance changes, I think having each dimension on its own thread has greatly helped overall performance, even in this situation. Garbage collection is very much improved. Server uptime is very much improved.

As result, I purposefully stopped deleting the Mineshaft.dat files (I did this automatically when the server would start, but I disabled it). However, when I stopped deleting Mineshaft.dat, performance slowly dropped over the days until I had to resume deleting the file once again.

So, the performance changes actually mask part of the issue, perhaps to the point that only very large worlds experience this problem to a crippling degree.

Can this be confirmed?

Again, I think MC-33086 is the root cause for this:

No it cannot be confirmed at this time, while it seems to be a known issue the exact root cause is not known.

"while this issue with mineshaft.dat may be causing these other issues, I still need to track down exactly why the file is becoming bloated in the first place. Unless there are comments that I'm not seeing, that question is still very much up in the air and will be a bit of a bear to track down."

I will mark it as community consensus though as I'm on board with you guys on several of the major server-related issues.

If you do a memory dump *after the server has been running a while to allow places to be explored*, you will notice the object representing the mineshaft.dat file can use several gigabytes of RAM.

16 Megabyte mineshaft.dat file (gzip compressed as most minecraft files are)
= 100 megabyte uncompressed mineshaft.dat file
= over 2 gigabytes in-memory object

What this means for me, with a server with -Xmx2G set, is that the CPU is spiking and the server is lagging because of continual full GC runs because the object is taking up space. This would seem to be the root cause of this bug, as well as a major contributing factor to MC-33086 (though I notice someone is also providing evidence that changing how the file is saved can help for that)

To be clear, I'm not suggesting a memory leak per se. If I load the save 16 meg data file using the python NBT library, it consumes over 2 gigs of RAM. Ditto NBT explorer, based on .NET. There's something common to how all three implementations store the object in memory that's causing this bloat.

I think it qualifies as a memory leak. Not just your garden variety memory leak. It's a memory leak that is persisted to disc.

The NBT format definitely got several issues. A few days ago i found another one, which I document here:
https://github.com/Poweruser/MinetickMod-1.7.2/wiki/Heap-dump-analysis:-Redundant-String-objects

Two more memory related issues i can think of right now:

• Each NBTTagCompound node object is initialized with a HashMap, to provide a comfortable mapping of string names to its sub nodes. The problem here is that these HashMaps have a default capacity of 16 fields, whereas the compound node that holds the information about a single mineshaft, has only 5 subnodes (ChunkX, ChunkY, id, a list of children (structure pieces), bounding box). And the different structure pieces have between 4 and 8 nodes. So, the HashMap could be built with a initial capacity of 8 instead of 16 and a load factor of 1 (or somewhere close, for them not to be resized to 16 again). But this fix is not future proof. Once there are compounds with 9 nodes, we're back to 16 again, as in the current HashMap implementation the capacity must be a power of 2 (2, 4, 8, 16, 32, ...).

• The amount of HashMaps that accumulates is insane as well. Example with a 1,17MB Mineshaft.dat file (compressed):
  This file has 708 recorded mineshafts. Each mineshaft has somewhere between 50 and 220 structure pieces (just a quick check in the NBTExplorer. Lets take 100 here, which is a optimistic average. Most are above 100, only a few are below. This makes a estimate of 70800 HashMaps. Additionally to that you have the HashMap-Entry objects which are about 6 times the number of HashMaps: ~425000 (nodes of a structure piece: 4 - 8)

Edit: The NBT format works kinda well for loading/saving files and after that immediately discarding it (like its done with region files and chunks [yes, they are nbt files as well]), while praying that the object is still in the young generation space of the heap, so the frequent garbage collection there can pick it up. If the object made it to the old generation space in the mean time you're screwed.
But it's not suitable for storing something permenantly in memory.

Anthony, your information is incorrect. A memory leak by its very definition is a program retaining allocated memory that is no longer referenced, such that the memory is never deallocated across the runtime of the program. That is most assuredly not what is happening here. I suggest reading Poweruser's description of the issue, as he is 100% correct.

It is by definition not a memory leak, as all of the memory in question is in active use by the NBT system. Mineshaft.dat effectively contains a massive NBT structure containing data on every structure piece of every mineshaft in the entire world, which is highly inefficient in terms of memory usage, but is not a memory leak by any definition currently in use in the programming field.

Poweruser, thank you for the analysis. While we currently have no ETA regarding a resolution for this bug, your information is very helpful in explaining the root cause of this issue.

Poweruser, I've been testing with the following Java options, which give me the best possible up-time, even with theses non-memory-leak issues:

java -server -Xmx4G -Xms512M -XX:PermSize=128m -XX:MaxPermSize=256m -XX:InitiatingHeapOccupancyPercent=80 -XX:+UseG1GC -XX:+UseFastAccessorMethods -XX:+UseStringCache -XX:+OptimizeStringConcat -XX:+UseBiasedLocking -XX:+UseNUMA -Xnoclassgc

The thing is, *because* this is extreme bloat and not a memory leak, you can just power through by just ensuring you give it enough memory. If you know your mineshaft object is going to take 2 gigs, allocate 2 gigs more heap. CPU problem goes away (because it doesn't have to run the GC constantly to keep the server running). The long save times would remain, though. Thanks Mog for acknowledging Poweruser's analysis, it's reassuring to know you're aware of the root cause of these symptoms, even if the solving it is still ongoing.

1.7.5 still uses a lot of CPU. I was hoping for improvements but i'll stay with spigot for the moment. Vanilla still uses 4x more CPU than spigot...

using the "nogui" option fixed the Cpu usage in 1.8 for me, but this must be relatet to another case.
.the exe Version for Windows didnt work fine(to much cpu usage), but the Java Version runs better, even with this "bug".

Still a problem in 1.8.3.

Why was this marked as cannot reproduce? Here's the steps to reproduce:

1. Start a new Minecraft server
2. Run, fly or teleport around to generate a lot of chunks
3. Restart the server
4. Note the CPU and RAM usage
5. Shut down the server
6. Delete Mineshaft.dat
7. Start the server
8. Note the CPU and RAM usage
9. Compare the before and after CPU and RAM usage

The more chunks you generate, the greater the difference will be, and the weaker the hardware it's running on, the fewer chunks you will need to generate before you start noticing the lag.

Is this still an issue in the most recent versions (currently that is 1.10.2, or 16w42a) of Minecraft? If so, please update the affected versions and help us keeping this ticket updated from time to time.

Yes, this is still an issue; in fact it is even worse than it was before. Please don't resolve first and ask questions later. There's definitely no such sense of urgency here. Did you not notice that this issue tends to get ignored for chunks of time lasting longer than an entire year?