Items cannot be removed from Item Frames by punching if the gamerule doEntityDrops is set to false. Steps to reproduce:
Create a world.
Place an item frame and an item into the item frame.
Punch it and notice the item is removed.
Do /gamerule doEntityDrops false
Repeat steps 2 and 3 but notice that the item cannot be removed. However, the item popping sound can still be heard.
Duplicate of MC-130558