Xbox UI / notifications cause severe fps drop when using Vibrant Visuals

When Vibrant Visuals are enabled, any part of the Xbox UI will cause the game to drop fps. For example: When using the Xbox keyboard to write in chat or on a sign, when I receive an Xbox notification in the bottom of my screen or when I open the Xbox guide. This fps drop will last about 20 seconds after the UI / notification is no longer on my screen.