Villagers store gossips for all entities that attack it, rather than just players entities. This adds (what I assume to be) unnecessary data for the entity
Steps to Reproduce:
1. Spawn villager
2. Check villager's gossips with `data get entity [TARGET] Gossips`
3. Summon zombie to attack the villager
4. Re-check villager's gossips with `data get entity [TARGET] Gossips`
Please check if this is still an issue for you in 1.16 Release Candidate 1 or later.