Iron Golem Stays Hostile in Peaceful

I accidentally hit my iron golem at my village, I didn't want to die so I switched the difficulty to peaceful. When I got out of the menu the iron golem was still trying to hit me, it didn't do any damage but I think the only way I can get it to stop is killing it.

See video attached