Ladders affect Spectators

I expected to not be affected by the ladders. just go inside the ladder as if you were in another game mode and shift. You should be able to sink, but you do not.