Target selectors can use the "m" selector argument to limit selection by game mode, but there is no value to select only players in Spectator mode.
To Reproduce: Run the following command:
/testfor @a[m=spectator]Expected result: List of spectators were shown
Observed: The error "Game mode 'spectator' is invalid' was shown instead
no command can test for m=spectator. not sure why.. I tried to use it for /tellraw @a[m=!spectator] but it just says "Game mode 'spectator' is invalid". I instead made 2 sets of command blocks for m=creative and m=survival to work around this.