It doesn't let me hit the allays

I think this sounds like intended behavior. When given an item to hold allays cannot be directly harmed/punched by their owners.

Allays are immune to player attacks if they are holding an item, this is intentional.