Other keys on hotbar won't work in inventory

I set the key "6" on "button 4" on my mouse and "9" on "button 5". When I'm in my inventory, I can press for example "1" while my mouse is on an item. The Item goes to Slot 1 in my hotbar. But when I do it with "button 4" or "button 5" it won't work. Button 4 and 5 work when I'm not in my inventory and want to switch to my water bucket or so, but not in the inventory.