Potion of Invisibility only works for Ghasts

When using an invisibility potion, even without armor, mobs still attack me like they usually do. The only mobs this potion works on are ghasts and spiders.