Game freezing briefly after tabbing out and then back into the game

The game freezes whenever I go out of it and then back in (using alt+tab on windows). The freeze lasts for about 5 seconds and it gets really, really annoying.

This effects all versions of the game; vanilla releases, prereleases and modded versions such as optifine. 

I have reinstalled minecraft by compleetly removing the .minecraft folder and letting the game reinstall by launching the game with the launcher again, downloading the game from scratch. This did not fix the issue.

The game never crashes. It only freezes when tabbing back into it. Never experienced a crash from this.