Saying I’m in the menus when I’m in a multiplayer game preventing some people from joining my world

 I’m playing with my friend(1) who’s on the Xbox one and I’m on the switch and he is able to join and then after some time it kicks him and he can’t rejoin even if I invite him and it shows that I’m in the menu and my other friend(2) hasn’t had that problem at all. And it does the same thing whenever I try to join my friend(1) it won’t let me join even if I was invited. This problem only happens whenever I’m playing with an Xbox one player on the switch and it works perfectly fine when I am on the Xbox one playing with another Xbox one player.