Can't place torches from offhand when no tool/block in main hand

The torch flashes on the block clicked but disappears immediately. The number of torches in the offhand decreases as expected. However when removing them from that slot, the number returns to the original amount. If all torches are "used" this way, the icon disappears, however clicking on the slot in the inventory brings them back.