There seems to be an FPS drop whenever you are in an Xbox party with a friend. I have tested this numerous times and seems to happen nearly every time it seems that the main issue is whenever the Xbox menu is being used. The FPS will drop immediately and then it will go back to normal but when in an Xbox party, it is permanent until you leave the party. The FPS will go from 60 FPS no drops to an immediate below 20 FPS or steady 20. I do not know the issue but I know how to reproduce it so I felt like it’s a bug hence the bug report.
This issue is being tracked at MCPE-199201.