Large FPS drop (When f3 display is open)

Not sure if this would be called a bug but either way it needs some serious optimisations.

Firstly, I recorded these stats on max settings, everything at max but gui scale (Normal) and clouds (Off). I was also looking straight down.

When I open the f3 display I suffer a loss of around 700fps (keep in mind i am looking straight down). The pics below were taken with fraps.

I was expecting a bit of a drop, but not one of that scale.