Breaking blocks in adventure mode?

I was in adventure mode and was only able to "break" 1 singular block, only with my first inventory slot when I was holding polished diorite, which was the block I was breaking. And when I did break it, it instantly went back to normal. Switching gamemodes didn't help either. It also kept its breaking animation even after taking my crosshair off of the block.