Not that I want to jump into the massive pile of reports but I will confirm that Xbox Series X is no stranger to these performance issues. Like I stated in my original post, it could be caused be the change in api. That being from DX12 to DX11. I could get a recording of the issue if necessary. The lowest frame rates (25fps is the lowest) are in forests or other biomes with heavy foliage.
This issue is still not fixed with the latest release, I found that jumping and moving the camera slowly seems to show the worst of this bug. Also it's been over a month and not a peep about this.