Iron Golem won't attack player if player attacks Villagers

When I am in survival mode, I punched a Villager and the Iron Golem did not attack me when it is suppose to attack the player if the player attacks Villagers