Doing some more testing, this seems to be fixable if you delay sending the response packet from the server by a second.
This issue has become worse in 1.16.100, the menu now doesn't show at all.
I have edited the issue to follow the format and contain a bit more information, thanks for the response.
The session directory the game reaches out to returns the session and you can join from the xbox app on windows without issue.