The method com/mojang/blaze3d/vertex/PoseStack#scale is broken in some situations:
1. If one or three of the scales are negative, the normal matrix will not be scaled correctly. The fix for this is to take the absolute value (Math.abs) of the argument passed into the net/minecraft/util/Mth#fastInvCubeRoot method. Additionally, net/minecraft/util/Mth#fastInvCubeRoot does not work for negative numbers, but fixing it does not solve this issue because it causes the matrix to be inverted.
2. If all scales are equal and negative, the normal matrix will not be scaled correctly. For this case, all normals should be inverted, but in reality nothing happens. The fix for this is to return after the normal matrix is inverted (normal.mul(-1.0f)).
Linked issues
relates to 2
Comments 4
In vanilla, the conditions for invalid values are usually met only when the model view PoseStack is scaled, but this stack's normal matrix does not matter since it is not uploaded to the shader or used in any other way.
The only places where the invalid values are actually used are in SignEditScreen#render, LoomScreen#renderBg, LoomScreen#renderPattern, and GameRenderer#renderItemActivationAnimation. GameRenderer#renderItemActivationAnimation is the only one that produces a visual effect. Right after being revived by a totem of undying, the totem in the activation animation will have lighting that does not transition smoothly as it spins. Applying my suggested changes fixes this.
Relates of MC-176864?
MC-176864 is caused by a series of model matrix transformations reversing the vertex order for all item model quads. Minecraft renders items (that are not translucent block items) with culling enabled, meaning that quads only render on the side from which the vertex order is counter-clockwise. Since the order is reversed in the opposite hand, all quads render from the wrong side.
From my testing, however, it does seem like the lighting bugs described in that issue are directly related to this issue. The lighting was fixed once my suggested changes were applied.
Does this affect anything in vanilla gameplay?