I have the exact same problem. I am the Realm owner and appear to be the only person on my Realm with the issue. Other xbox players connect just fine, even players from my household.
It happens to me too. I am the Realm owner and the only player on my Realm with the issue, but not the only Xbox player.
The only solution I have found is hard restarts, sometimes taking as many as 5 - huge pain being the owner who pays for a service and can't log in to my own Realm. I tried logging in and out, quitting the game and restarting, uninstalling, nothing works but the hard restarts and even then its not guaranteed.
They were teleporting to me on login too. Ended up killing my parrot just because it was such an inconvenience so it might as well not be in the game at this point since I can't use them properly.