I believe this is really a conflict between ladders and the mechanics of flying. Considering you can stop on a ladder while crouching, and flying down is ALSO by crouching. The ladders and vines are taking hold of the player, thus making flying down in jungles, or flying down 1 block holes with ladders in them near impossible.
Related issues
is duplicated by
relates to
Comments
I have seen this since before 1.2. The fix would do just disregard ladders and vines while flying, or have a different key for flying down and crouching.
Also happens in 1.4.7 prerelease
This is still an issue as of 13w09c.
Confirmed for 1.7.4 and 14w04b

Confirmed for
14w30c
14w31a
Minecraft 1.8-pre 1

This also affects spectator mode.
qmagnet, that's MC-46626, which was fixed in 28b.

Well wha'd'y'know! I was not aware it was fixed!

Confirmed for 1.8.1.

Confirmed for
15w37a
Confirmed for 15w46a
Confirmed for 1.9-pre1
Confirmed for 1.9

Confirmed for 17w14a

Confirmed for 1.12-pre3.
Can confirm for MC 1.12.1.
Confirmed for 1.13.1.

Affects 19w14b
Affects 19w45a
This problem still occurs in version 1.14.4
Affects 1.15-pre1
Still in 20w17a against twisting vines.
Confirmed