Days ago I had that happen to me with my brother also when we were playing together on his survival world and not only the following items makes it weird how our character holds it but sometimes one of our characters would act like a ragdoll while walking or sprinting as seen on our 3rd person POV. And most importantly we both didn't have our Resources Pack turned on.
Even on Xbox One is having the issue too.
If Mojang reads this, I would like to tell them that I have great connection including my NAT set to Open on Xbox and the internet is not the main cause of the issue, it's their game that automatically prevents me to join my brother's world just by displaying a "Unable to connect to world" message even though we both are in the up-to-date version (v1.17.11 Hotfix), we both have Xbox Live Gold, we have our Xbox profiles added, and everything. Why can't they just yet see?
_