The server can indefinitely hang when stopping

When shutting down the server, it can completely hang while saving worlds.

Reproducing this issue can be difficult due to it being caused by a race condition.
I had most success when throttling the server, loading a lot of chunks and running /stop while the chunks were still being generated, although this was still quite rare.

The expected behaviour is that the server should always shut down gracefully when running /stop

The actual behaviour is that rarely the server completely hangs when shutting down, and never finishes saving properly.

This hang happens due to the server calling MinecraftServer#waitUntilNextTick() while waiting for the ChunkMaps to finish working on shutdown. waitUntilNextTick() runs the following code: this.managedBlock(() -> !this.haveTime()) which blocks the thread until haveTime() returns false. haveTime() returns this.runningTask() || .... While debugging I observed runningTask() to always be returning true despite that not actually being true, which caused the hang.
runningTask() returns this.reentrantCount != 0, and reentrantCount is only ever modified in:

// net.minecraft.util.thread.ReentrantBlockableEventLoop
@Override
protected void doRunTask(final R task) {
	this.reentrantCount++;
	try {
		super.doRunTask(task);
	} finally {
		this.reentrantCount--;
	}
}

I presume the intention is that doRunTask() is only ever ran on the “main” thread, and thus the code doesn’t account for this being able to be run by other threads. But if it is able to be run by other threads reentrantCount can be concurrently modified and since ++ and -- are not atomic it’s possible for an update to be swallowed and overwritten by another thread which can cause reentrantCount to no longer equal 0 after all tasks are run which propagating back up the call tree causes haveTime() to always return true and thus the server to hang indefinitely.

Now as for why doRunTask() is ran off the main thread - BlockableEventLoop#execute schedules tasks if this.scheduleExecutables() returns true, and otherwise calls doRunTask():

// net.minecraft.util.thread.BlockableEventLoop
@Override
public void execute(final Runnable command) {
	R task = this.wrapRunnable(command);
	if (this.scheduleExecutables()) {
		this.schedule(task);
	} else {
		this.doRunTask(task);
	}
}

BlockableEventLoop#scheduleExecutables returns true if the current thread is not the main thread, but MinecraftServer overrides this method to make it return super.scheduleExecutables() && !this.isStopped(), so if MinecraftServer#execute is called off the main thread while the server is stopping then the task will not be scheduled and instead doRunTask(). Attached also (in one of the comments) is a stack trace where I had a breakpoint on doRunTask() while stopping the server for calls that happened when !this.isSameThread() confirming that it can indeed be called off the main thread.