Light tickets are released too early

The issue described in the following is the root cause of MC-214793. While it has only become visible in practice recently due to MC-224893, I think this is still an important concurrency issue on its own and hence deserves its own report.

During worldgen, special light chunk tickets are created just before the light generation stage, in order to keep the chunk and its neighbors loaded until initial lighting is finished. Or at least this seems to be the idea behind this mechanism. However, the ticket is removed right before starting the actual initial lighting, rather than waiting until it is finished.

net.minecraft.server.world.ServerLightingProvider.java

public CompletableFuture<Chunk> light(Chunk chunk, boolean excludeBlocks) {
  ChunkPos chunkPos = chunk.getPos();
  chunk.setLightOn(false);
  this.enqueue(chunkPos.x, chunkPos.z, ServerLightingProvider.Stage.PRE_UPDATE, Util.debugRunnable(() -> {
...
super.setColumnEnabled(chunkPos, true);
...
this.chunkStorage.releaseLightTicket(chunkPos);
  },
    () -> "lightChunk " + chunkPos + " " + excludeBlocks
  ));
  return CompletableFuture.supplyAsync(() -> {
chunk.setLightOn(true);
super.setRetainData(chunkPos, false);
return chunk;
  },
    (runnable) -> this.enqueue(chunkPos.x, chunkPos.z, ServerLightingProvider.Stage.POST_UPDATE, runnable)
  );
}

This is of course too early and leaves the possibility for the neighbor chunks to unload before the initial lighting has finished, causing lighting glitches at the chunk borders (the chunk itself is actually kept from unloading due to the pending light task). These border glitches can happen even in 1.16.

Note that chunks below the FEATURES stage are considered opaque by the lighting engine. Hence, even though the chunk itself is kept from completely unloading, its ticket level might drop below FEATURES stage which would make the chunk opaque to the lighting engine (during initial lighting), hence causing dark chunks as observed in MC-214793. In 1.16, this issue does actually not occur since chunks won't be demoted once they finish a generation stage (more precisely, the respective Future for that generation stage stays valid, which is precisely what the lighting engine checks). However, this is no longer true in 1.17 due to MC-224893 and hence the issue actually becomes visible.

In order to provoke this issue, one can traverse the world at high speed, e.g., using spectator mode (with very high speed) or tp commands in quick succession. Only a sparse set of chunks will make it to the LIGHT stage during this traversion, other chunks will already abort generation after the FEATURES stage. Furthermore, at the time these LIGHT stages are processed, the player is already far away, leaving the ticket level below the FEATURES level. Both of these facts together, sparseness of light tickets and chunk ticket levels being too low when the initial lighting is processed, make the above explanation applicable. And indeed one can observe completely dark chunks when visiting the traversed region again. Note however, that this might require some attempts due to the random nature of this bug. (In my test cases these were however quite reliably reproducible).

Note that while MC-214793 mentions corrupted features that often occur in conjunction with dark chunks when flying in spirals, no corrupted features were observed with the above steps. In conclusion, I think that the above steps really isolate the lighting bug itself, whereas the situation described in MC-214793 is a convolution of the pure lighting bug and other phenomena of MC-224893, see the latter for a small discussion.

Finally, let me mention that I did move the releaseLightTicket(...) call to the POST_UPDATE phase and no further dark chunks were observed, as expected. However, I did observe some small lighting glitches at chunk boundaries. This was in fact expected, since the light ticket only loads neighbors into LIQUID_CARVERS stage, whereas they should really be in FEATURES stage for the lighting engine to interact with them. So this is actually yet another issue with the light ticket mechanism (in 1.17).

Remarks

Let me also remark that this issue is actually an instance of a more general concurrency issue present in the chunk loading code. As mentioned in the beginning, it could already happen in 1.16 that neighbor chunks unload before the initial lighting is completed, causing light glitches at the boundaries, due to the light ticket being removed too early. This effect is not new to 1.17.
More generally, the chunk loading/worldgen code does not take any precautions to guarantee that a chunk stays loaded once it is passed to any worldgen step. Once a Future completes, the chunk is passed to the next generation step (on the worldgen thread) which can then hold on to the reference indefinitely. Independently, on the server thread, the chunk then may be unloaded before this next generation step does its work, so any progress after this point would be lost.
Similarly, the light ticket only handles the case of initial lighting. Any further updates after the initial lighting do not take any measures to keep the chunk and its neighbors in a state where the lighting engine can interact with them. So, any light updates after the LIGHT generation stage (but before promotion to FULL stage) might get stuck randomly due to chunks being unloaded or being demoted from the correct state.

As a solution, one should consider to implement a mechanism similar to the light tickets for general interactions during worldgen.

Best,
PhiPro