When falling from any height in survival, your fall damage is randomized. It should be a standard amount, with no randomization.
To reproduce:
1.Equip feather falling 4 boots
2.Make a pillar 25 blocks tall, walk off, and see how much damage you get dealt.
3.Regenerate, your health and repeat as many times as you like. Mark down how much damage you take each time, and you will see it's anywhere from 2 1/2 to 6 hearts.
As you can see this is quite the issue with game balance, and needs a quick fix!
this happens to me all the time
I have a raid farm and before I got elytra I would scaffold up and jump down from y=120-130ish to ground level (y=64ish)
I was always holding a totem in my hand and have feather falling 4 on my boots
occasionally I would use a totem… more often I would end up with 4-8 hearts damage… and every once in a while I'd take 2 hearts of damage
then sometimes when I fell onto the pillager outpost (which is y=87ish) I'd use totems more often than when I fell onto the ground which is, again, about 20 blocks lower