Can’t break banner after placing on map

I am encountering an issue where, having placed a banner and attaching a map to it, I cannot break the banner. No matter where I click it, it breaks the blocks around it. I’m playing in Survival mode, and I renamed the banner in an anvil prior to this.