If and item such as a Diamond Sword gets any AttributeModifier, it will "reset" its Attack Speed, making its Attack Speed like in the middle of Diamond Hoe and hand.
/give @p minecraft:diamond_sword 1 0 {AttributeModifiers:[{AttributeName:"generic.attackDamage",Name:"generic.attackDamage",Amount:10.5,Operation:0,UUIDMost:1234,UUIDLeast:-452334}]}
When applying the AttributeModifiers tag to any item, all default AttributeModifiers disappear, this is the intended behaviour, so you can have for example a sword without any attack speed delay, this has been this way since the tag was introduced.