Replacing item to quickly the player keeps the attributes from the item. Clearing or logging off doesn't fix this either. the only fix is removing the system and placing the item back into off hand then removing but only if it's the same uuid. If lost item with uuid you need to use an alt program to remove the attributes.
To Recreate:
/give @p minecraft:diamond_axe 1 0 {AttributeModifiers:[{AttributeName:"generic.movementSpeed",Name:"generic.movementSpeed",Amount:-1,Operation:0,UUIDLeast:340481,UUIDMost:754575,Slot:"offhand"}]}
scoreboard objectives add test dummy
scoreboard players set @p test 1
(Repeating) /testfor @a[score_test_min=1] {Inventory:[{Slot:-106b,id:"minecraft:diamond_axe"}]}
(Chain Conditional) /replaceitem entity @a[score_test_min=1] slot.weapon.offhand air 1 0 {Inventory:[{Slot:-106b,id:"minecraft:diamond_axe"}]}
(Chain Conditional) (This does nothing just gives the axe back) /replaceitem entity @a[score_test_min=1] slot.hotbar.0 minecraft:diamond_axe 1 0 {AttributeModifiers:[{AttributeName:"generic.movementSpeed",Name:"generic.movementSpeed",Amount:-1,Operation:0,UUIDLeast:340481,UUIDMost:754575,Slot:"offhand"}]}
Also I suppose you don't need a scoreboard but it's what I used.
I can confirm for 1.9.