Cannot play with specific friend

I can play with other friends in multi-player, but when I play with my brother, the game will pop me out to leave the game, even have a good WiFi connection