If you mine a block close to the ladder, you'll get a mining penalty

This report is invalid as your game was modified when experiencing this issue. The statistics displayed in the upper left corner of your screen indicate that you're running a non-vanilla instance of Minecraft, and bug reports are only valid if the problem can be reproduced in vanilla.