All people with extreme examples of this problem (i.e. fps dropping to single digits) have been using 32 bit Java (see the thread in the description).
In all cases when these people changed to 64 bit Java they had a noticeable performance improvement - even with CPUs that don't support AVX (for example the above-mentioned LeslieGilliams).
For anyone using 32 bit Java, moving to 64 bit Java gives a significant reduction in the FPS drop caused by this problem.