[MC-72390] Rcon is not thread-safe

Use of Rcon to execute commands that modify server state causes a race condition that may result in a java.util.ConcurrentModificationException in the main server thread if a command modifies data during a server tick.

Conversely, executing commands via Rcon that iterate over internal data structures (such as /save-all) may cause a java.util.ConcurrentModificationException in the Rcon thread if the main thread modifies those structures (see MC-68374).

Update 2016-11-08

During reproduction of this bug, it was observed that introducing a small delay (as low as 0.003 seconds) between each command sent via rcon appears to greatly reduce the chance-per-command of the bug being triggered. However, with zero delay between commands being sent, the chance of the bug being triggered is significant, with usually less than 100 commands being enough to trigger the bug.

End update

In the following example, an external script was being used to run commands. A ConcurrentModificationException was thrown when the Rcon thread /summon ed a Creeper and added it to the world's entities list, while the main thread was simultaneously iterating over the same list due to a command block executing /kill @e[type=MinecartTNT] . (Full crash report is attached)

[2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: effect RobotoRaccoon 11 60 100 true
[2014-09-26 21:14:21] [RCON Client #3/INFO]: [Rcon: Given Resistance (ID 11) * 100 to RobotoRaccoon for 60 seconds]
[2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: playerinfo RobotoRaccoon
[2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: summon Creeper 2209 86 501 {Fuse:0}
[2014-09-26 21:14:21] [RCON Client #3/INFO]: [Rcon: Object successfully summoned]
[2014-09-26 21:14:21] [Server thread/ERROR]: Encountered an unexpected exception
u: Executing command block
        at net.minecraft.server.MinecraftServer.z(SourceFile:603) ~[minecraft.jar:?]
        at po.z(po.java:228) ~[minecraft.jar:?]
        at net.minecraft.server.MinecraftServer.y(SourceFile:531) ~[minecraft.jar:?]
        at net.minecraft.server.MinecraftServer.run(SourceFile:447) [minecraft.jar:?]
        at java.lang.Thread.run(Unknown Source) [?:1.7.0_55]
Caused by: java.util.ConcurrentModificationException
        at java.util.ArrayList$Itr.checkForComodification(Unknown Source) ~[?:1.7.0_55]
        at java.util.ArrayList$Itr.next(Unknown Source) ~[?:1.7.0_55]
        at aqu.a(SourceFile:2230) ~[minecraft.jar:?]
        at ah.a(SourceFile:433) ~[minecraft.jar:?]
        at ah.b(SourceFile:136) ~[minecraft.jar:?]
        at ab.a(ab.java:71) ~[minecraft.jar:?]
        at aqf.a(SourceFile:97) ~[minecraft.jar:?]
        at auo.b(SourceFile:53) ~[minecraft.jar:?]
        at qt.a(SourceFile:527) ~[minecraft.jar:?]
        at qt.c(SourceFile:199) ~[minecraft.jar:?]
        at net.minecraft.server.MinecraftServer.z(SourceFile:599) ~[minecraft.jar:?]
        ... 4 more
[2014-09-26 21:14:21] [Server thread/ERROR]: This crash report has been saved to: /data/minecraft/18-temp/./crash-reports/crash-2014-09-26_21.14.21-server.txt
[2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: summon Creeper 2209 87 501 {Fuse:0}
[2014-09-26 21:14:21] [RCON Client #3/INFO]: [Rcon: Object successfully summoned]

Reproduction

Create a new Minecraft server instance.
Enable Rcon on the default port (25575)
Set Rcon password to "test"
Connect a Minecraft client to the server.
Move the player to x=0, z=0 (or use /spreadplayers 0 0 1 1 false @a) to ensure this location is loaded.
Use Python to run the attached rcon.py script, which will send commands to the server (spawning entities at x=0 z=0)
The server should crash in under 60 seconds (usually only a few seconds).

I have successfully reproduced a ConcurrentModificatonException crash using this method. Due to the nature of this bug (race condition), it may take some time before the crash occurs.

Linked issues

is duplicated by 6

MC-68374 ConcurrentModificationException when performing save-all via rcon Resolved

MC-74058 Legit Rcon crashes minecraft Resolved

MC-82752 When using rcon, I sometimes see this show up on the console. Resolved

MC-91817 multiple rcon commands crashes server Resolved

MC-147977 Exceptions are randomly thrown while saving from RCON Resolved

1 more links

relates to 3

MC-509 Exception in server tick loop :: ConcurrentModificationException Resolved

MC-87863 RCON uses TCP as though it's datagram-based Reopened

MC-46850 Possible Race Condition with RCON Resolved

Attachments

1.10.8 Server Log 2016-11-08.log

crash-2014-09-26_21.14.21-server.txt

crash-2014-09-27_03.28.36-server.txt

crash-2016-03-03_16.41.02-server.txt

crash-2016-11-08_17.52.23-server.txt

rcon.py

Comments 38

TerrorBite 2014-09-26T17:40:32Z

Attached Python script used to reproduce the bug. Connects to Rcon on port 25575 with password "test" and issues a rapid series of /summon Fireball commands (server despawns the fireballs immediately).

TerrorBite 2014-09-26T17:41:10Z

Attached crash report generated by reproducing the bug using rcon.py

Amn3zik 2014-10-14T15:06:06Z

Minecraft Crash Report ----
// Shall we play a game?

Time: 10/10/14 11:04
Description: Exception in server tick loop

java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
at java.util.HashMap$KeyIterator.next(HashMap.java:1453)
at qn.a(SourceFile:166)
at net.minecraft.server.MinecraftServer.z(SourceFile:615)
at po.z(SourceFile:305)
at net.minecraft.server.MinecraftServer.y(SourceFile:531)
at net.minecraft.server.MinecraftServer.run(SourceFile:447)
at java.lang.Thread.run(Thread.java:745)

A detailed walkthrough of the error, its code path and all known details is as follows:
---------------------------------------------------------------------------------------

– System Details –
Details:
Minecraft Version: 1.8
Operating System: Linux (amd64) version 2.6.32-042stab092.1
Java Version: 1.8.0_20, Oracle Corporation
Java VM Version: Java HotSpot(TM) 64-Bit Server VM (mixed mode), Oracle Corporation
Memory: 2473019160 bytes (2358 MB) / 6242500608 bytes (5953 MB) up to 10522722304 bytes (10035 MB)
IntCache: cache: 0, tcache: 0, allocated: 12, tallocated: 94
Profiler Position: N/A (disabled)
Is Modded: Unknown (can't tell)
Type: Dedicated Server (map_server.txt)

Amn3zik 2014-10-14T15:09:35Z

Minecraft Crash Report ----
// Shall we play a game?

Time: 10/10/14 21:58
Description: Exception in server tick loop

A detailed walkthrough of the error, its code path and all known details is as follows:
---------------------------------------------------------------------------------------

– System Details –
Details:
Minecraft Version: 1.8
Operating System: Linux (amd64) version 2.6.32-042stab092.1
Java Version: 1.8.0_20, Oracle Corporation
Java VM Version: Java HotSpot(TM) 64-Bit Server VM (mixed mode), Oracle Corporation
Memory: 1308330000 bytes (1247 MB) / 4080271360 bytes (3891 MB) up to 10522722304 bytes (10035 MB)
IntCache: cache: 0, tcache: 0, allocated: 12, tallocated: 94
Profiler Position: N/A (disabled)
Is Modded: Unknown (can't tell)
Type: Dedicated Server (map_server.txt)

TerrorBite 2014-10-14T15:19:02Z

@Amn3zik Out of interest, what were you attempting to do via Rcon that triggered that crash?

28 more comments

Amn3zik 2014-10-14T16:16:56Z

I was just doing a / give object, but we test multiple commands with a friend are all planted serveur at one time or another.

Sorry for my english is a google trad

TerrorBite 2014-10-14T16:20:28Z

Thanks for the info!

Amn3zik 2014-10-14T16:29:17Z

this bug is really annoying 😞

Greener 2015-10-28T00:01:04Z

This bug affects the latest snapshots, could we get an affects 15w43c? Maybe they can fix this for release?

Greener 2015-10-28T00:14:30Z

Minecraft Crash Report ----// I'm sorry, Dave.

Time: 10/27/15 11:12 PMDescription: Exception in server tick loopjava.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429) at java.util.HashMap$KeyIterator.next(HashMap.java:1453) at ln.a(SourceFile:180) at net.minecraft.server.MinecraftServer.D(SourceFile:626) at lb.D(SourceFile:314) at net.minecraft.server.MinecraftServer.C(SourceFile:546) at net.minecraft.server.MinecraftServer.run(SourceFile:450) at java.lang.Thread.run(Thread.java:745)
A detailed walkthrough of the error, its code path and all known details is as follows:---------------------------------------------------------------------------------------

– System Details --Details: Minecraft Version: 15w43c Operating System: Linux (amd64) version 3.13.0-29-generic Java Version: 1.8.0_45, Oracle Corporation Java VM Version: Java HotSpot(TM) 64-Bit Server VM (mixed mode), Oracle Corporation Memory: 338889136 bytes (323 MB) / 1495793664 bytes (1426 MB) up to 1743257600 bytes (1662 MB)
JVM Flags: 0 total; IntCache: cache: 0, tcache: 0, allocated: 12, tallocated: 94 Profiler Position: N/A (disabled) Player Count: 0 / 20; [] Is Modded: Unknown (can't tell) Type: Dedicated Server (map_server.txt)

Greener 2015-10-28T00:17:19Z

and in the console:

00:15:01] [RCON Client #207/ERROR]: Failed to save chunkjava.util.ConcurrentModificationException at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1207) ~[?:1.8.0_45] at java.util.TreeMap$KeyIterator.next(TreeMap.java:1261) ~[?:1.8.0_45] at lq.a(SourceFile:691) ~[minecraft_server.15w43c.jar:?] at lq.a(SourceFile:674) ~[minecraft_server.15w43c.jar:?] at ash.a(SourceFile:232) ~[minecraft_server.15w43c.jar:?] at ash.a(SourceFile:102) [minecraft_server.15w43c.jar:?] at lp.b(SourceFile:147) [minecraft_server.15w43c.jar:?] at lp.a(SourceFile:166) [minecraft_server.15w43c.jar:?] at lq.a(SourceFile:885) [minecraft_server.15w43c.jar:?] at ay.a(SourceFile:36) [minecraft_server.15w43c.jar:?] at j.a(SourceFile:81) [minecraft_server.15w43c.jar:?] at j.a(SourceFile:64) [minecraft_server.15w43c.jar:?] at lb.a_(SourceFile:588) [minecraft_server.15w43c.jar:?] at nd.run(SourceFile:80) [minecraft_server.15w43c.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]

Greener 2015-10-28T17:09:35Z

Yup, present in 15w44a

Greener 2015-10-30T19:20:34Z

Yup, 15w44b

Greener 2015-11-05T23:56:02Z

Yup 45a

Greener 2015-11-12T16:39:06Z

~~47a. Yup~~

Oops.. 46a

WubbiConcepts 2015-11-22T09:01:38Z

Happened to me in 15w47c as well.

TerrorBite 2015-11-22T10:34:27Z

It's been over a year since this bug was reported, and the issue persists. It's perhaps worth noting that a popular third-party server project fixed this same issue over three years ago.

Greener 2016-02-16T16:55:54Z

Affects 16w07b

Greener 2016-02-17T16:50:22Z

Affects 1.9-pre1

Greener 2016-02-18T19:07:47Z

affects 1.9-pre2

Greener 2016-02-29T21:25:46Z

affects 1.9

TerrorBite 2016-03-06T08:47:40Z

It is unlikely this bug will be fixed without special attention, since the observed behaviour is an artifact of the way that Rcon has been implemented (as a separate listener thread). The only way to resolve the issue is to implement special handling of Rcon commands by queueing them for the main thread to handle (and likely blocking the Rcon thread until the result of the command is available), rather than naively executing them directly on the Rcon thread where they can interfere with concurrent operations on the main server thread.

Vincent Loiacono 2016-05-04T20:28:39Z

I use vanilla votifier and send 4 commands through Rcon, crashes are random . sometimes I can spam rcon with commands and it's fine , usually when no one is on the server, other times, all it takes is one set of commands to Rcon and the server crashes with the ConcurrentModificationException error message. My server is a vanilla server, now running 1.9.3 pre 3. I would like to see this issue resolved. thx. if you need any crash reports from me, feel free to ask.

bob 2016-11-06T11:09:49Z

Is this still an issue in the latest snapshot 16w44a? If so please update the affected versions.

This is an automated comment on any open or reopened issue with out-of-date affected versions.

TerrorBite 2016-11-08T06:54:42Z

Updated rcon.py test script that uses snowballs instead of fireballs (due to some changes to entity handling that cause summoned fireballs to vanish before they spawn)

TerrorBite 2016-11-08T07:07:10Z

This issue is confirmed to affect 1.10.2. Crash report attached. Server log attached.

During reproduction of this bug using the attached rcon.py script, it was observed that introducing a small delay (as low as 0.003 seconds) between commands appears to greatly reduce the chance-per-summon of the bug being triggered. However, with zero delay between commands being sent, the chance of the bug being triggered is significant, with usually less than 100 entities being successfully summoned before the crash occurs.

The current test script summons 20 snowballs every 2 seconds, meaning the server usually crashes within 10 seconds of the script being started.

Zedadias Wick 2016-11-10T12:14:11Z

Confirmed for 1.11-pre1; modified provided rcon script to summon snowballs, server crashed almost immediately.

crash log: https://pste.me/#/uuRpZ

Tron 2017-01-06T21:20:40Z

Unfortunately still a thing in 1.11.2; had a script to run a bunch of RCON commands from a text file and this is the crash I get.
http://pastebin.com/raw/3uUR383P

Tron 2017-04-27T04:47:32Z

Still appears to be an issue in 17w17a, However I've never had a crash when there are no players on the server, Perhaps the crash only happens when players are online?

pokechu22 2017-04-27T22:10:58Z

From what I recall of the issue, it's only a problem if you attempt to run commands through rcon; just being passively connected won't cause any problems. (However, this isn't a ticket that I've looked into deeply)

jonathan2520 2017-04-28T14:55:17Z

Yes, only while running commands. I guess if no players are on the server, entities aren't active, preventing a race condition there. Not that I've tried this recently or extensively. I ran into it when I was trying to load large structures into a live world programmatically. Using commands like setblock and fill in/around water causes a crash very quickly. Must be all the block updates of water trying to flow everywhere while blocks are simultaneously being replaced by rcon.

TerrorBite 2017-04-30T05:24:49Z

Yes, the specific race condition used as an example in this bug report occurs in the entity processing code. Rcon commands that manipulate entities will cause a crash when the server is also processing entities at the same time, since Rcon and the server operate on different threads. If there are no players on the server and thus few or no entities are loaded, the server will be spending little or no time processing entities, avoiding the race condition.

However, entity processing is only one area that is affected by the root cause of the issue, that cause being the Rcon thread modifying server data in a non-thread-safe manner. As @jonathan2520 has discovered, a crash will also occur if the server thread is handling block updates for water flows while Rcon commands are also trying to modify blocks. This has nothing to do with entities, but the cause of the crash is ultimately the same.

Ultimately, the fix for this bug must involve one of two solutions: Either

utilizing locks with critical Minecraft data structures, to prevent concurrent modification, or
pushing Rcon commands into a queue from where they will be popped by the server thread, so that ultimately the commands are run on the server thread.

pokechu22 2017-04-30T05:37:54Z

pushing Rcon commands into a queue from where they will be popped by the server thread, so that ultimately the commands are run on the server thread.

Worth noting that this is already what is done for the normal protocol, and that there already is a thread-safe list of pending commands; Rcon just doesn't use it.

jonathan2520 2017-04-30T17:57:07Z

Minecraft's rcon implementation is nothing but a quick hack. Every shortcut that doesn't prevent it from working at all was taken. You may remember MC-87863 as well. I say they should just rewrite it from scratch. They wouldn't be throwing much away anyway.

Fenhl (Max Dominik Weber) 2017-11-05T04:05:50Z

Unfortunately, setting a 0.003 second delay is not a sufficient workaround to stop our hourly world backup script from occasionally crashing the server.

Yass B. 2018-07-19T22:28:11Z

Confirmed for 1.13

Yass B. 2018-07-27T18:29:48Z

Does someone know how to fix this ?

rasensuihei 2019-05-19T01:30:36Z

It's still happening on 1.14.1 😞
I guess almost commands are unsafe except "help".