Placing wrong block after using hotkeys to move item in hotbar (Creative)

While I'm in creative and I hover over an item slot and push a number to move it to that slot, (It has to swap places with another block) it places down the block that was previously in the slot