XP orbs sometimes won’t collect

I have a zombie farm, so I wait for a little bit to let the zombies spawn in and drop down to the kill chamber. Now usually there’s a ton of zombies, so I kill them the XP drops out but I don’t collect most of it. This happened before the 1.16 update, it just wasn’t as severe happening very rarely. It’s really annoying as I can’t fix my tools.