Entities' collection order and ticking order get scrambled on chunk reload

Almost every redstone contraption breaks when chunks are reloaded. This happens for a plethora of reasons, two of which are explained here:
(They are heavily intertwined, so they are reported together.)

• Entity collection order (the order in which the entities are stored in their chunk sections) is scrambled on reload.
  This only happens to passenger entities, because when unloading/loading passengers, the game iterates over the passenger tower recursively, completely disregarding the order in which these entities are stored in their chunk sections. Because of this, the order in which the vehicle and its passengers are stored in the chunk is shuffled after a reload (which is undesirable).
  To be more specific, this occurs in the loadEntitiesRecursive() and PersistentEntitySectionManager.processChunkUnload() methods (and possibly in some other methods that we may have overlooked).
  To clarify, the expected behavior is that the order in which entities are stored in their corresponding chunk sections (i.e., the order in which they are interacted with by pistons and such) should remain unchanged upon a chunk reload.

To replicate and observe this collection order issue:

1. Open the provided world download.

2. eleport to the setup using the following command:
   /execute in minecraft:overworld run tp @s 949.51 99.00 1004.41 223.20 5.25
   There will be 3 buttons (one on a piston and two on command blocks).

3. Press the button on the command blocks marked as “summon.” It will summon two villagers - the top one is riding the bottom one - and they are stored in that chunk section in the order: top villager, then bottom villager.

4. Press the button on the piston. Observe that the lamp does not turn on.

5. Reload the chunk with the villagers (for example, by relogging).

6. Press the button on the piston again. Now observe that the lamp does turn on. This happens because the villagers' collection order was reversed.

7. If the steps above are unclear, they can be viewed here:
   https://youtu.be/FrQoGXeSezE?si=9ryH40ccdSLNtduG&t=245 (note the timestamp)

This issue can be fixed by rewriting the passenger loading/unloading code so that it respects the order in which the entities are stored.
However, special care must be taken with edge cases where the passenger and its corresponding vehicle are in different chunk sections, or even in different chunks. For example, if the vehicle is unloaded while its passenger is far away, the passenger will vanish from the world — even if it was directly in front of a player and may have been interacted with. Here is an example of such an edge case: https://youtu.be/EPQfs0QAlrs.

Please be aware that only the relative collection order between entities within the same chunk section matters - the relative order between entities in different chunk sections should not be changed.
Currently in the collection order all entities from one chunk section go before all entities from another chunk section (with the chunk sections themselves ordered directionally).
It is important to preserve this order, as numerous redstone contraptions already rely on it. (From mass item crasfters to tnt cannons, this behavior is surprisingly overused)

• Entity ticking order (the order in which entities are ticked) gets scrambled on reload.
  This happens for the following two reasons:

  • When a chunk is reloaded, the entities are not added to the ticking queue in the same order they occupied before the chunk was unloaded. Instead, they are added in the order of their corresponding chunk sections, from bottom to top — all entities from the bottom chunk section are added before all entities from the chunk section above.
    
    To replicate and observe this ticking order issue:

    1. Open the provided world download.

    2. Teleport to the setup using the following command:
       /execute in minecraft:overworld run tp @s 954.04 160.00 999.01 90.75 14.85
       There will be a contraption that detects the ticking order of entities.

    3. At your feet, there will be a button. Press it - this will spawn two armor stands (note that they are in different chunk sections).

    4. ress the noteblock near the piston once. Observe that the lamp does not turn on.

    5. Reload the chunk with the armor stands (for example, by relogging).

    6. Press the noteblock near the piston once again. This time, observe that the lamp does turn on.
       This happens because the armor stands’ ticking order was swapped, due to the bug explained above.

    7. If the steps above are unclear, they can be viewed here:
       https://youtu.be/FrQoGXeSezE?si=hAtt1ReFrwS-OKci&t=109 (note the timestamp)

  • When two entites, “a” and “b”, happen to be in chunks “A“ and “B“ respectively, if the entities are ticked in order (a,b), and only chunk A is reloaded, then the entity ticking order will become (b,a) which is obviously problematic.
    
    This issue is rather self-explanatory. To replicate and observe this ticking order issue: load two chunks, A and B, spawn an enity a in chunk A and an entity b in chunk B (in that order). Log the ticking order using your preferred method and observe that a ticks first, then b ticks second. Now reload only chunk A. Log the ticking order again and observe that b now ticks first and a ticks second.

Fixing both of these ticking order issues is only truly possible by adding a long integer field to each entity that stores its ticking order.
(Currently, Block Ticks do exactly this to preserve their ticking order between reloads; it is called ScheduledTick.subTickOrder in the Mojang mappings.), It might seem that the id field could be used for this purpose, but beware, the id field itself cannot be used to determine the entity ticking order for the following reason:

Currently, there exists a very tmc-sensitive behavior that is important to preserve:
When an entity is moved out of a lazy chunk into an entity-processing chunk (usually by a piston or through a .move() call), it is added to the very end of the ticking queue (i.e., its ticking order is reassigned). To preserve this behavior, the ticking order has to be re-assigned specifically when, and only when the entity is moved from a non-entity-processing chunk into an entity-processing chunk via a move() call.

However, the id field should not be reassigned as it governs other important mechanics that rely on the id remaining constant. Therefore the ticking order field must be separate from the id.

Also, beware of this nuance:

Currently, if an entity is spawned within the entity phase (i.e., spawned during another entity’s tick), it will only start ticking in the next closest entity phase, not the current one. It is important to preserve this behavior, as it is very tmc-sensitive.

Here are a couple of related bugs:
MC-263030 and MC-299681 (cannot attach a link for this one because its private)

If anything mentioned in this report is unclear and requires clarification, please ask in the comments. These issues are very tmc-sensitive, so any careless changes can and will cause a lot of redstone contraptions to needlessly break.
This video can also be referenced, as it covers the above-mentioned issues with in-game examples:
https://youtu.be/FrQoGXeSezE

Finally, if a decision is made to fix this bug, we (tmc) propose that we provide an exact fix that we know resolves these issues without breaking any redstone contraptions or behaviors.
This will help avoid inevitable miscommunication, saving us from a lot of re-reporting and the developers from a lot of re-writing.