I have the exact same issue. I am playing on the Series X, and I am trying to connect to my son's world on the Nintendo Switch.
His world shows up under the "Joinable Friends" browser. However whenever I try to join, it just hangs on the "Locating Server" portion of the "Generating World" window. I have tried waiting for a significant amount of time, and it never allows me to connect on my end. The strange thing is on his end, it does show my name, and it will even say I've disconnected when I hit cancel on my end. His world isn't even very big, it's only like 5MB right now. NAT shows as open. I am pretty sure we were able to do this before. Perhaps it was a recent update that caused this?
This issue is still ongoing with version 1.19.60. I still cannot join a world hosted from a Nintendo Switch with a XBOX Series X. It is worse than before. Now it just times out when trying to connect.