Unable to play with specific friend

Me and one of my friends have tried for about 3 weeks to get into the same world and play together but every time there is a message that say unable to connect to world we have both tried restarting the game and our xbox's and the issue was still there we have also tried on difrent worlds as well as on a realm and nothing has worked we can join anybody else except for each other