Steps to reproduce:
use default 1GB of memory allocated
Create new creative world with customized world type with mega taiga hills biome (this is to ensure known, constant average memory usage per chunk)
Set render distance to 32
Press F3 and wait until MultiplayerChunkCache value no longer increases (should get to about 3800-4000)
Spawn a sheep and hit it
Save and exit
Without restarting Minecraft, load that world again
Move a few hundred blocks away from spawn
After a while you should notice the game freezing very frequently for a fraction of a second, caused by being close to running out of allocated memory (depending on exactly how far away you move, you may run out of memory and crash Minecraft). Sometimes hitting an entity again makes the issue go away, sometimes it doesn't.
Why this proves there is an issue?
Because being close to full memory usage means the JVM will try to do full GC, removing all stale objects. So if even after that, memory usage is still high enough to frequently trigger full GC, there muct be more memory actually being used by something. Heap dump can confirm that there actually are 2 WorldServer objects ("oo" class in 1.12.2 obfuscated Minecraft).
Most likely cause:
I initially found this issue with my mod, and I couldn't find anything in heap dump pointing to my code. The only GC roots seemed to be related to a class MCP calls EnchantmentHelper and it's inner Iterator classes.
After testing it in vanilla, I found that it's actually vanilla issue. The issue seems to be that the inner Iterator classes hold references to Entities, potentially long after the world the entities are in has been unloaded (until they are replaced by entities from a new world, which most of the time happens when the player hits an entity, I wasn't able to figure out why it fixes the issue only sometimes).
Comments 5
The memory leak from WorldGenBigTree is actually fixed by MinecraftForge, that's likely why I didn't notice it when first testing it and analyzing with forge. Only later I confirmed the issue with vanilla and found that hitting an entity doesn't always fix it.
And, confirmed for 18w15a... However, it's a lot better there; the world generator rewrite that happened in 1.13 seems to have fixed the leaks related to worldgen (at least I didn't have any leaks from it), leaving only this as the leak. (As per forge's MCPConfig, in 18w15a WorldServer is sh and EnchantmentHelper is ate; ate$a was the only thing holding the old sh instance).
Having difficulty reproducing in 18w20b as reloading the world turned the instance unresponsive, can someone else try?
This was almost fixed in 1.13-pre4, but a typo causes the fix not to work; in the method MCP calls applyArthropodEnchantments (however, I'm suspicious of the accuracy of that name), the damage iterator is set, but the attacker for the hurt iterator is cleared. This typo is on line 224 of the actual source.
Confirmed; there definitely is such a leak with those iterators.
Unfortunately, I can also confirm and explain why punching an entity doesn't always fix it: it seems like other stuff also can hold a reference to the world. In my case, I first tried going into a superflat world, and that didn't solve the issue because a reference to the world was still held via
WorldGenBigTree.world(fromBiome.BIG_TREE_FEATURE). There are probably other such leaks too.