This happens to my Mending Enchanted tools as well on Windows 10, it's annoying enough that I'm not playing until it gets fixed, also playing on a realm, haven't tested on local world yet
Also happens when fishing
I'm currently testing a fortune I diamond shovel on a new map, and have only seen 2's, 3's, and 4's
I know this happens to me when I'm playing in my realm, when Mining with my Diamond Pickaxe(Mending,Efficiency V, Fortune III, Unbreaking III) and fishing with my enchanted Fishing Rod(Luck III, Lure III, Unbreaking III, Mending). And, similar to josh's experience, this happens when the tools collect XP for Mending. I tried testing a normal fishing rod on a local world, but had no luck reproducing the shaking. I haven't tested with the other tools, either in the Realm, or on local worlds.
Also, I'm playing the Win10 version of Minecraft