Being close to the end island causes out of memory and file descriptor exhaustion

Being in a certain distance to the end island causes a massive memory leak.
Recreation can be seen in

This happens since 18w44a.

This first happened when @unknown and I tried to recreate MC-140808 and MC-137467.
We were able to identify the render distance and the distance to 0,0 as important factors.

Steps to reproduce

1. Create new World (creative, cheats enabled)

2. Set render distance to 8

3. Go to the end /setblock ~ ~ ~ end_portal

4. Fly to avoid falling into the void and teleport to /tp 140 70 0

5. Press F3 and watch the memory usage grow to 100% in a couple of minutes

Alternatively use the world download

with render distance 8.

Issues

Besides the massive memory leak, I also noticed random processes on my system shutting down.

Further investigation showed that Minecraft spawned thousands of threads each keeping hundreds of file descriptors alive, completely exhausting the system resources.

@unknown noticed that the snooper shows thousands of chunks being loaded in the overworld as well as the end, which may be related.

In 1.13.2 the snooper shows only about a fifth to tenth of that

.
However, interestingly this was already the case in 18w43a/b/c in which the memory leak doesn't happen.

Cause

The cause seems to be an armada of world generation threads spawned by the dragon fight manager.

The fight manager makes sure a 17x17 area around 0,0 is loaded before it summons the dragon.
Judging from the number of CompletableFuture instance created within a couple of seconds, it seems like it creates one for each chunk in each tick, but for some reason doesn't succeed loading them.
Note: The dragon does not spawn, presumably because the area never gets loaded. Also see MC-137467.

Stack trace of the main culprit.

Stack trace:
	java.util.concurrent.CompletableFuture.doThenCombine(CompletableFuture, BiFunction, Executor)
	java.util.concurrent.CompletableFuture.thenCombine(CompletionStage, BiFunction)
	m.b(CompletableFuture, CompletableFuture)
	m$$Lambda$2093.153347328.apply(Object, Object)
	java.util.stream.ReduceOps$1ReducingSink.accept(Object)
	java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Consumer)
	java.util.stream.AbstractPipeline.copyInto(Sink, Spliterator)
	java.util.stream.AbstractPipeline.wrapAndCopyInto(Sink, Spliterator)
	java.util.stream.ReduceOps$ReduceOp.evaluateSequential(PipelineHelper, Spliterator)
	java.util.stream.AbstractPipeline.evaluate(TerminalOp)
	java.util.stream.ReferencePipeline.reduce(Object, BiFunction, BinaryOperator)
	m.b(List)
	uj.a(bag, int, IntFunction)
	uj.a(bag, int, bqe)
	tz.a(bqe, uj)
	tz.a(uj)
	ue.a(uj, tz)
	ue$$Lambda$2087.569814135.accept(Object)
	java.lang.Iterable.forEach(Consumer)
	ue.a(uj)
	uj.l()				// WorldGenWorker.runTasks() ?
	uj.b(int, int, bqe, boolean)	// WorldGenWorker.addTicket() <<< Probably a good point to start looking at
	uj.a(int, int, bqe, boolean)	// WorldGenWorker.genChunk()
	bas.a(int, int, bqe)		// createChunk()
	bas.d(int, int)			// loadChunk()
	brj.c(int, int, int, int)	// DragonFightManager.loadArena()
	brj.a(int, int, int, int)
	brj.a(brj, int, int, int, int)
	brj$b.a()
	brj$b.a(brj$b)
	brj.b()				// DragonFightManager.tick() <<< Pretty sure the check could be better on top level too
	brk.l()
	uk.m_()
	net.minecraft.server.MinecraftServer.b(BooleanSupplier)
	net.minecraft.server.MinecraftServer.a(BooleanSupplier)
	doq.a(BooleanSupplier)
	net.minecraft.server.MinecraftServer.run()
	java.lang.Thread.run()

The underlying issue might relate to other currently reported world generation bugs and memory leaks, such as MC-138114 or MC-139930.
However I wasn't able to find a post that matched the description closely enough to be sure.