Lagging and not responding in multiplayer worlds

My gf is trying to get me to play in her world (on Switch) and I am entering with PS5. It is really slow to load, lags a lot, and is unresponsive. You cannot open doors, chests, enter boats.. Also the game throws you back. If I am walking somewhere, it will automatically throw me back several meters, constantly. If I am cutting a tree, there is no wood to collect. We have both paid money to have cross-play, but it is infuriating to try and play together.