Cannot "test" for slot 99 (off hand)

If you try to assign a score with specific NBT, you can't do:

/scoreboard players set @a offHand 1 {Inventory:[{Slot:99b,id:"minecraft:banner"}]}

This works with every other slot, just not slot 99.