Large camera lag

I keep having these periods where my framerate will drop. unexpectedly when I press Shift + F3 I notice that camera it taking up the majority of the pie chart. No matter what I change my setting to or even if I change into a void world this camera thing causes me a lot of lag. Sometimes my frames will go down as far as 60. The game doesn't feel smooth when the fps is 60 or higher when this is happening. When I go to window mode the goes away significantly. And this lag problem I'm having isn't constant, it comes and goes.

Just found out that toggling fullscreen fixes it for the most part.