Elytra glide effect doesn't stop when removed from chest slot while falling

When you are gliding with elytra, if you remove the item from your chest slot, the gliding effect does not stop. You keep gliding until you hit the ground.

To recreate, teleport up far enough that it will take a minute or 2 to reach the ground, start falling with Elytra equipped, change gamemodes, while gliding remove elytra, and you will keep gliding.

EDIT: I shortly after found that you don't start gliding if you equip Elytra shortly after you start falling