when i'm repairing my Mending pickaxe at my exp farm. when my armor is on most of the exp goes into my bar and not into my pickaxe. when i take my armor off. it all flows directly into my pickaxe and only a small amount goes into my exp bar. main or off-hand doesn't matter.
Does your armour have mending? If so, that's why this is happening. The wiki has a good explanation how mending works.