Can't play with my friends online because the game doesn't wants me to

I added my neighbors as Xbox friends so we can play together without being at each other's house, but now it won't work, their accounts are not restricted to multiplayer, neither mine is, but when we try to join each other's worlds without playing in LAN it won't work. It starts loading and then: "could not connect to the multiplayer world" and that's it. We can only play by LAN but we can't be at each other's house all the time and this is been there for more than one year; the same problem with different people as well