I am now experiencing this issue on 1.20.4 and 24w07a.
For me, whatever the state of the cursor is when entering fullscreen, the mouse stays like even when swapping from first person/ to menus.
The same thing happens if the cursor is hidden when entering fullscreen, the cursor is invisible when navigating menus.
Additionally, when multiple monitors are connected, there the mouse is always hidden.
I am currently using the Dev Windows 11 (Version 10.0.26058 Build 26058) insider release with an NVIDIA 2070 running version 551.61.