World loading stutters, stops completly

If you set the render distance to/below the view distance in the server.properties file, the world loading stutters outside the already loaded chunks or mostly stops completly. In singleplayer everything is fine. It just happens on a server.