Fail to join multiplayer game

Symptom:
When joining Multiplayer game created by another Windows 10 player over Xbox Live (not LAN), it stuck at "Generating World Locating Server.." and responded with "Unable to connect to server" after a while. The World is about 200Mb in size. Same case even repeated the joining for 5 times.

Supplementary Info:
1. Joining the same save (i.e., World) from LAN has no issue
2. Joining another smaller save (~15 Mb) from the same host via Xbox live has no issue.
3. After performed step 2 in the above, things would become normal (i.e. the reported symptom is gone). You are able to join the concerned 200Mb world.