I don't get the issue on my newer XBox One Series S (2020), but I had the issue on a XBox One (2013) and a friend of mine had the issue on their XBox One S (2016). Perhaps the issue is specifically with older versions. It was always a tad slow, but something about 1.19 makes it take 3 times as long to load (not an exaggeration).
Something similar is happening for Minecraft VR. It is unplayable because the trigger button makes the hotbar switch back to another item. It's like it gets stuck on one item. Change it to a sword to try and fight and when you hit trigger to swing, it goes back to that same inventory item.