Mobs can still track you and hurt you even if you have the invisibilty effect active. They will track you regardless of if you hit it before or not, or if it was previously tracking you.
Test:
1. Summon Zombie.
2. Use invisibility splash potion.
3. Wait a while or punch zombie and it will follow and damage you. Notice they can track you even if you do not hit it before.
Apparently this was fixed according to MC-18284 but it still is occuring...
Linked issues
Comments

It happens when you don't hit it as well. It will eventually begin tracking you regardless of if it previously was following you or if you hit it before.
Are you sure you didn't just get too close? Mobs will attack you if you get within a block or two of them, even if you are invisible.
Not intended. Dupe of MC-10901

Well that's odd, how come MC-18284 was 'fixed' then?
MC-18284 was fixed as that bug only affected mobs who got an AI upgrade.
If you hit a mob, it will track you. That's intended.