Endermen keep chasing me after becoming neutral

After provoking an Enderman at night, he turns neutral at daytime. That's normal. But the Enderman still teleports near me all the times. Somehow it's still chasing me. It's very annoying. Sometimes they just pop up and scare me. I guess this could be a bug!