Nether loading issues but only after visiting the nether once already.

I noticed issues like mine but not quite the same. I am able to load into the nether once and leave without problems. But when I try to back into the nether it just says building terrain then never loads in. I have waited for it to load for over 5 minutes. I end up going to the xbox homescreen and force closing the app just to get it to work one more time. While I was waiting for my side to load my portal was blown up and my character took damage, so the game is still going but I'm stuck on the loading screen. I have tried restarting my xbox one but the problem persists.