Server: Mob pathfinding induced massive lag in any versions 15w49a and later

Its listed as resolved, when will this resolution be put into a snapshot?

MC-93093 is resolved as "Working as Intended" but on a review list.
Cannot tell when it's reviewed.

Kumasasa,
Forgive this if it sounds rough, but did you review my log snip? It isn't resolved, I just tried the snapshot. This isn't a log from weeks ago, its today with the release. It isn't fixed or resolved in the current snapshot.

Sure I did read your log snippet. Therefore I resolved as dupe of MC-93093.

Ok. I'm not entirely proficient at submitting bug reports, I've only did a few. Let me see if I follow.

The bug still exists in the most recently released snapshot, and you are labeling it as a dup for a previous snapshot, and tagged it as Resolved and working as intended.

So, massive block lag is the planned outcome of the newest snapshot and is indeed, "working as intended".

If I am wrong in my assessment, Did I incorrectly submit? I don't understand how Mojang can confirm a bug in a snapshot, but label it resolved, dup and working as intended despite evidence it isn't? Whats the point in continuing to report bugs?

Reading the comments from others regarding this, I see I am not the only person who has issue with the "Working as intended". I'm sure mojang is at least aware of the poor classifications.Thank you for your time.

MC-93093 was resolved by Mojang.
Neither other users nor we mods aren't lucky with that decision.

How can this be a duplicate of MC-93093? 15w47c was already spamming the console with that message, but it didn't have the huge lag spikes.

Everything past 15w47c is basically unplayable on a remote host. I haven't experienced those lags in singleplayer either, but I haven't tested much.

This definitely has NOTHING to do with MC-93093. It's more a duplicate of MC-94289, which I submitted myself.

I uploaded a 2.5 minutes console snippet of server running 15w51b. Just me standing idle doing nothing.

Ok, reopened.

I can also confirm the packet spam and lag. CPU usage shoots up to 100% during this.

I can confirm the lag, even in singleplayer in snapshot 15w51b. I use 2 tests: a redstone clock test and /debug. I have uploaded 2 /debug reports. The packet flood (I got already 150 packets per tick in SP) is maybe a side effect.

Edit: I have found out that this is the pathfinding algorithm that jams the game (from the first report). The first report is in an experimental world, the second is from a superflat world with nothing.

@theo Oh you managed to get it in singleplayer too? Nice find. I had the feeling it was related to pathfinding somehow. Pretty sure it's the new zombie AI. That's what made the game crash in 49a, it was quickly patched in 49b but I guess it was just some sort of temporary band aid solution.

How do you dump profiles like these? Is it built-in to the debug profiler in-game?

/debug start and /debug stop to do the profiling

@theo

Thank you. I'm used to /timings in Spigot and didn't even know vanilla had this! Pretty useful. 😃

I don't think the lag is only from zombies AI, because in the superflat world, even in daytime, there is significant lag. And even in the experimental world, zombies are killed every 6-7 seconds. But I think this is actually from pathfinding (in general).

Edit: I create a world with nothing, just a platform. The lag doesn't occur, even with redstone contraptions.

You're right. For some reason I thought the changes to pathfinding AI since 15w47c only concerned zombies, while it was global. I follow change logs closely, I have no clue where I got that idea. Perhaps in video somewhere... :/

I even think that the snapshot 15w51b aggravates the issue because this snapshot fixes MC-94419, which fixes a crash involving pathfinding and chickens. The issue may be global. And also, I noted that even killing all entities isn't useful once the lag is installed. However, I don't experience any significant lag before 15w51b, maybe because of the patch.

I too can confirm this issue on a new server ( ubuntu 14..04 LTS ) the server details ...

Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz
2 120GB SSD
32GB Ram (only using 4GB for MC server)

Running snapshot 15w51b

With one player the server seems to do fine. Running around 20% CPU usage and 6% ram, although tons of console spam ... (player) is sending move packets too frequently ... between three to eight lines per second ranging from 6 packets to 18 since last tick.

As soon as another player joins the server jumps up to about 60% CPU usage, ram stays about the same, but the CPU will eventually start spiking past 100% CPU usage and eventually crash after about 20 min of game play.

Here's the last server crash (from yesterday) if that helps. ... http://pastebin.com/kjBUsHtf

Let me know if there is something else I can provide as I am new to bug reporting.

Thanks ... Pops

Happens to me too on 15w51b. Sometimes the server tps inexplicably drops down to 11tps (while not under load) and otherwise seemingly under the same circumstances it can also operate at ~19.5tps. The packet warning spam, unusual lag, and crashes happen as well.

System details:
Windows 8.1 64bit
AMD FX6300 3.5GHz 6-core
7200rpm Seagate HDD was used to run the server (i/o queue was very low at times of crash- about >15% utilization and >20ms response time)
12GB ram (of which 4GB was allocated to the server)

Move packet spam unsurprisingly doesn't happen when nobody is moving, and crashes don't seem to follow a discernible pattern- sometimes it runs for days without a crash, sometimes it crashes half an hour after startup. I didn't feel like posting crash logs because people seem to have a really annoying habit of flagging them as duplicates of MC-63590, saying behaviour is intended. Completely misses the point of posting reports- we all know watchdog isn't broken, it's the circumstances that matter (and crashes are not intended behaviour).

Anyway, I digress; here are the most recent crash logs:

http://pastebin.com/GHs23NMA
http://pastebin.com/vbwvMBGf

Please do not repeatedly edit your comment, a notification goes out to everyone watching the bug every time you save. There is a preview button if you're concerned about checking the format.

Alright, sorry for the notifications spam.

On the other hand, here's another bug report on 15w51b where something trips the watchdog, in case it'd be of help to anyone: MC-95174 (of course, it gets flagged as a duplicate). Changing the watchdog settings doesn't fix whatever is causing the issue- don't quite understand why crashes are labelled as intended behaviour.

I am certain if Unknown2k told us, "We are running 16w01 for 10 min. to get profile data for Mojang, please bear with us," we would do our best to provide "real world/ in the wild" data to confirm there really is a problem, and help isolate and get a specific target to do the bug fix. The problem should seriously affect Realms as a use case.

I am getting massive lag on a 16-core server where I've given Java 8Gb of ram, while it is admittedly amplified terrain, there are only two of us playing, and it only needs one of us for it to start lagging. This is 15w51b. I am seeing the same console spam and getting watchdog crashes.

This server lag is real and very intense, to the point of eventually crashing it. I can pretty much reproduce it. I have a villager trapped a fenced pen and a valid village 40 blocks from him. As soon as he starts pathfinding to the village, without being able to reach it, the cpu usage skyrockets, packets get delayed more and more causing unbearable server lag.

This is on a dedicated server. I haven't tried in single player. I'll try to find a conclusive way to reproduce it.

The server lag is real, but I haven't yet observed the watchdog crash. I killed almost all mobs (hostile mobs and villagers) and prevent mobs spawning, the lag is significantly reduced (but not completely: I still got 17 ticks per second).
Edit: There is a workaround: kill every hostile mobs and then set the gamerule doMobSpawning to false.

This remains unfixed. We can't use any snapshots above 47c. We run our server in Adventure mode at first, to control the block people spawn in on.

I think Mojang is aware of the issue and it's maybe why snapshots are delayed (I take account of the holidays). Snapshots must not be used in public servers because of the potential instability (this seems to be the case).

I'm a bit confused. How does Mojang get "under realworld use-case testing" to find and fix problems unless the software is getting "under realworld use"? In this case, the staff of the server is under no illusions about the stability of the server, and they are very diligent to communicate with us (the players) about the state of the snapshot and to suggest a Spigot server if the player wants "stability".

Or does Mojang not really care to have their server perform under "real use", preferring to leave good server performance to a third party? That doesn't make sense to me, given Mojang now offers Realms. I'm sure paying Realms customers would prefer not being the first to test 1.9 server "realworld".

Rather than saying, "Snapshots must not be used in public servers", perhaps it would be good to encourage use on the bleeding edge with a sense of proportion and perspective, and more than a bit of humor. We do that, we are consistent about it, and we really want to help build a better product. Waiting on Spigot to clean up Mojang's mess on the server side is really a poor answer.

Just a thought for consideration, since real world servers having real world issues are complex. I imagine a conversation like the following. I have no idea if anything mentioned here is, in fact, anything to do with anything. Just the issue of how complex all this can be, and the kind of patience and questions it will take to resolve the problem.

Mojang: Oh! There's your problem, you are protecting way too much area at spawn! That setting was never meant to be used that way. Lower it, and you'll be fine.
Snapshot: That's the area we need to protect while providing services to our player base you don't easily provide in Vanilla.
Mojang: Well, why don't you try Thing X to do that, and it will be easier on the server and work just fine.
Snapshot: We did do it that way up to the point where some idiots trashed the area because of a hack.
Mojang: Oh, well that hack was fixed in version 15w##a, a later snapshot than you are running.
Snapshot: We are not running it because the block lag was so horrible after version 15w47c made it unplayable with more than 'Y' players. So we update... how can we test to make sure this is really fixed under conditions we can control?
Mojang: I know the specific hack used and we have a test rig. With your permission, I can attempt to use it on your server after the update.
Snapshot: Ok, let's do this thing
— one test later —
Mojang: Oh! We did fix it, but this Z thing you are doing makes the fix not work properly. We didn't know that. Let's try ...

And so on. Real world conditions with "normal use" players including idiots who attack the server on a regular basis simply because it is there, it is popular, and it is well run. A perfect opportunity for Mojang to figure out what combination of real world issues require real-world settings that make the snapshots after 15w47c unplayable. I don't represent Snapshot, I'm just a player who prefers missing the latest bugfixes in updated snapshots to having our server unplayable, but that's like saying I'd rather have a kick in the rear to a kick in the groin.

@William (Andy) Smith

Anything beyond 47c is unplayable because of the big changes in "improved" mob pathfinding introduced in 49a. It crashed a lot and a "band-aid" fix was released in 49b. I say "band-aid" because, well, every time my server chokes up, the profiling always points to pathfinding eating up 99% of the game ticks. And I know pathfinding is broken simply by looking at mob behaviour. They can't even get to you if they have to climb anything oriented north-south. Baby zombies and endermen move like flies and are next to impossible to hit, and we can't spam them anymore, expecting a hit or two. So we die, no matter what, because them, they NEVER miss you.

And here comes the infamous "Don't discuss here" that I trigger every time I try to help... Note to this mod that I won't name: This is relevant to the issue!

@Early Reflections:
Another gripe I'd like to mention pertaining to this is that for every crash report submitted, despite a clear correlation between the crashes and the new snapshots, is that they invariably get marked was Working as Intended. I mean, sure, watchdog is working as intended, but the ticket isn't about the watchdog, it's about whatever is causing the crash- and a crash certainly isn't intended behaviour.

Perhaps people could be required to test the same thing on a stable release, to rule out the common issues and common causes- but if there's an automatically generated crash report on a snapshot version that hadn't been present before, it should be reported. It does not follow that crashes and their subsequent reports can only happen as a result of user error.

Hope this can get some attention. It is very hard to provide feedback on the new snapshots when they are basically unusable.

Confirmed in 16w02a.

It felt better for about 30 minutes then CPU and RAM meters skyrocketed and tps fell way down to around 12. This needs SERIOUS attention. The whole dev team should only focus on this single issue alone for the next snapshot. Keep the minor bugs, cosmetics and rendering triffles for dessert. The community NEEDS useable snapshots to help speed up remaining fixes!

I know they can be unstable and flaky, and I'm ok with that. But in my experience (since 13w01a), no snapshot in history were THAT bad, and for so long.

Mojang says that we should never use snapshot for live servers. I'm sorry but live servers running snapshots are required for real-world feedback. Here's a real-world feedback: everyone left my server during the holidays. I ended up all alone and most of my friends got mad at me, because I insisted on keeping 15w51b running during that time. I should have kept 47c, but then I couldn't report any issues because it's not the latest snapshot. They're not coming back. I'll test all alone from now on.

Maybe we have the same issue. Mine is MC-94684. This is caused by Commandblocks. If i use 30-40 Commandblocks, just repeating testfor @p oder some Particlecommands, i get this lags in less than a minute. With less commandblocks it needs longer. Its caused by repeatingcommandblocks, no matter what command they do.

Also confirmed for 16w02a

If this issue stays in 1.9 every customized Vanilla-Server and every Adventuremap ist unplayable!

We are at this point pretty sure that the major cause of lag is Mob AI. See earlier comments that point the finger firmly at "the pathfinding algorithm jams the game (from the first report)".

There may be a separate issue with command blocks, but this is unrelated.

Maybe there could be a correlation.

With great expectations come big disappointments. 16w03a. Tried it for only 3 minutes. The first few mobs that spawned brought my server down to its knees, 100% CPU, 100% RAM, 7 to 9 TPS. I could hear the machine scream for mercy! 😃

@unknown, quote from https://mojang.com/2016/01/minecraft-snapshot-16w03a/ :

Some things that were broken in the previous snapshot (16w02a) are not fixed or changed back in this snapshot, because we had to focus on few very important bigger tasks to work on before we can go back working on smaller changes. This is nothing to worry about and perfectly normal so close to finishing a release, so don’t get upset if you see things that are still in the same state as last week. We’ll take care of them soon and definitely before we call Minecraft 1.9 done.

@Kumasasa

I'm perfectly aware of that, as I've read it whole thing before doing anything else. The fact is, this is not an issue with the previous snapshot (16w02a), it's been a MAJOR issue in ALL snapshots since 15w47a.

Look at the fixes:

MC-779 - MINOR, cosmetics
MC-68383 - MINOR, cosmetics
MC-71006 - MINOR, cosmetics
MC-80826 - NOT URGENT, doesn't affect playability
MC-95539 - NOT URGENT, annoying but not a showstopper
MC-95541 - NOT URGENT, doesn't affect playability
MC-95612 - MAJOR but doesn't affect playability

On the other hand:

MC-89928 - VERY MAJOR, unaddressed for over 3 months. Makes the End inaccessible without cheats. Ironically, it's the new anti-cheat system that causes it.
MC-94438 (This one) - VERY MAJOR, unaddressed for almost 2 months. Seriously affects playability. TBH it's literally unplayable.

I'm not pissed off, I'm just being critical. I know I can be hard on you guys. I don't question your job, what you're doing is amazing and demanding, especially with pricks like me. But I definitely question the devs' sense of priorities. They seem to be doing what they please, regardless of what's important or not. But, oh well, it's just a game, and it's not my baby, it's theirs. I shouldn't be putting my heart into it, but I just love it too much.

Quote from Searge on reddit:

we deal with our bug list in the order that works best for us

and

Snapshots are not stable and are not meant to be playable versions of the game. I'm not sure why I even have to point that out.

I can understand Early Reflections impatience. I am so excited for the release and can´t wait much longer. 🙂 I realy hope, they will fix this and similar issues asap, cause i want to finaly test, if everything works. I know these are snapshots an their only purpose is to test the game, collect feedback and find bugs, but for a lot player its more than this. It´s the time for preparing for the release. I am doing lot of customizations for my server, so i can set up everything on release. (sry for bad english)

@redstonehelper

You DIDN'T have to point that out. Look at my post from jan 14th about minimum playability and real-world feedbacks. I'm tired of this being pointed out every single time. I KNOW IT, OK?

I accept flaws and bugs and stuff... that's why I try to help debug the thing. I just don't accept them lasting forever and never being looked into.

@Sebastian

I totally share your opinion and excitement. 1.9 will an awesome release, once it works!

"We will deal with our bug list in the order that works best for us"
It is possible that they may not currently know how or why this bug exists, and are just currently focusing on bugs that they know how to fix. Once those bugs are all out of the way, they'll turn to more complicated bugs like this one.

I think the main feel with this particular issue is that while snaps arent meant to be completely stable and are beta//bug testing with a majority of the larger servers of users sticking with 47c further and complete better testing and reporting of bugs maybe slowed. I understand mojang and the team have a way of doing things and they do well at it, but perhaps something from one of the mojang crew in here would appease the masses so to speak

@All: STOP.
This is not a discussion forum. For any meta discussion please use https://www.reddit.com/r/Minecraft

@Kumasasa

You're gonna hate me for this, if you don't already. Reddit is a cesspool. Everything not of the 1st page doesn't get any attention and everything is mixed together in a totally indecipherable mess. And on top of that, you redirect us to the wrong place. It should be https://reddit.com/r/mojira for discussing bugs.

If you really want to stick with Reddit, I'd suggest that for every single new bug report on the tracker, a reddit thread be created automatically for it. And on the bug tracker page, a direct link to that exact discussion thread. Otherwise any relevant discussion gets buried under pages of irrelevant threads, never to be seen or found.

The best idea would be to have your very own, properly organized discussion forum. But that won't happen. Am I the only one who finds it a bit ridiculous to not be able to discuss bugs on a bug tracker?

There is nothing against discussing bugs on a bug tracker, but any meta discussion about how and when Mojang is fixing the bugs does not belong here. So, if you have anything to contribute to this issue Massive Lag in any snapshots post-15w47c like confirming, steps to reproduce or similar, feel free to do it, otherwise simply don't do it.

I can also confirm this issue on my server, and in single player. It's been there for months, and has no specific cause, but CAN be pinpointed at least to a specific area. You can see my findings here: http://www.minecraftforum.net/forums/support/unmodified-minecraft-client/2603466-profuse-lag-extreme-ram-usage-and-console-error

I've been doing some experimenting to recreate this bug, and the easiest way to recreate it is to simply dump a bucket of water in front of a pack of zombies that are trying to path to you.
It seems that liquids cause a huge problem with path-finding.

Video: https://www.youtube.com/watch?v=lYkDAhZpUQs

I'm currently not sure if anything else causes problems with path-finding, but liquids are for sure one of them.

Interesting
I haven't really had much lag at all, even when playing on a LAN world with my friend, and pretty much the only place I've had lag in vanilla survival is when fighting endermen in the End. I first thought that was because of their high follow ranges of 64 blocks (I am pretty sure large follow ranges is another problem with path-finding, which would mostly be prevalent in custom maps), but with Jono's comment, I think it's because I normally place down water to keep them from hitting me.

In my case, I can assure you that it's definitely not liquids or anything to do with them. or at the very least, they aren't the major cause of lag. Using the same effected region file I provided in my Minecraft Forum post, I replace all liquids with solid blocks, and the problem still persisted. In addition, deleting all mobs that have complex pathfinding, custom or otherwise, also did not help. It's really beginning to get on my nerves how ethereal this bug seems to be. _

I tried messing around with your posted region file.

I haven't been able to pinpoint the source of the lag for sure, but it isn't due to path finding.
It doesn't help that the offending tick in the debug profiler is "unspecified". (levels/map/tick/tickPending/ticking/unspecified)

On a fresh map created with the latest snapshot, this tick uses only 0.20%.
With your region file, that tick uses 68%!

I can kill all entities with /kill @e, and some of the lag goes away as well as the console spam. However, not all of the lag is gone and the problem is still with the unspecified tick, whatever that is used for...

I want to try and see if I can recreate this with a fresh map. Unfortunately, problems like this have been caused in the past by using older maps with newer versions of the game, which really sucks if that is the case here.

Well I'm glad somebody with more debug knowledge than I has had a chance to look at it! You've definitely pinpointed more than I could, so that's a step in the right direction. On a semi-related note, I can probably say for certain that it's not an issue with it being an old map. While it IS true that the terrain here is absolutely ancient by Minecraft standards, (This was the first world I ever generated in beta 1.5_01!) in the interest of keeping it clean, corruption, and border-free, I have many times completely regenerated the map with MCEdit, including one time recently under the assumption that this issue was a corrupted chunk. It was not.

Essentially, the world itself is a 3008x3008 box covered on all sides by a stone wall, and 40 chunks of barrier blocks replacing air going inwards from there to prevent the wall being visible. Players can still get to the outside via the rail system, but not by land. ANYWAYS, the point of all of this, is that from time to time, I use MCEdit to prune the outlying chunks, and copy the entire map as a schematic, only to reimport it into a blank world. So while the map itself is technically very old, from a real world standpoint, the data that makes it up has been completely regenerated at least 10 times since then.

I've had some problems with lag and the watchdog timer in the snapshots. And what I noticed was that some redstone circuits would stop working before the crash. The most common being tripwire getting stuck in the triggered state, removing the tripwire and string doesn't fix it. the server needed to be restarted to reset the tripwire, it also sometimes fixes itself after a while.

Java arguments also have an affect on the amount of lag, I used -verbose:gc to see what the garbage collector was doing and found that GC1 seems to work the best, tested parallel and CMS as well. Also the lowest cpu usage was with java 9 early access builds.

The java command I'm currently using for java 9 is -Xmx3G -Xms2G -XX:-DisableExplicitGC -Xnoclassgc -XX:+UseG1GC -XX:+UseStringDeduplication -XX:InitiatingHeapOccupancyPercent=70 -verbose:gc

When testing with java 8 adding -XX:+AggressiveOpts lowered cpu usage and the frequency of skipped ticks.

Setting the watchdog timer to 5min also has mostly stopped it from triggering.

The server we are using is a core-duo with 4G of memory on windows 10.

Confirmed in 16w04a.

From the looks of things, Mojang is fixing bugs in two sets of order; categorically, and numerically. Unless it's just their way of listing fixed bugs on the blog, they pick a theme for the snapshot, and fix each applicable bug in numerical order. Furthermore, for whatever reason they seem to keep the more severe bugs for last. At this point, I'm honestly not even expecting a fix anymore until pre-release. I just hope it gets assigned to somebody before then, because despite what I'm assuming should amount to "community consensus", this bug hasn't even been officially CONFIRMED yet..

@Pixie: I tried your JVM arguments with Java 9 x64, and my game, (along with my entire PC) pretty much froze, so I couldn't confirm or deny your findings. Weird since while we share the same amount of ram and OS, my CPU is three times as powerful as yours. o-O

@unknown, the performance fixes and increments are always done last in development, so don't complain that it isn't assigned yet.

The "snapshot theme" is mostly a joke, they just fixed one boat issue, and the others were linked as related and thus were fixed as well.

Here's 122 seconds of console output (truncated for space):

28.01 16:43:50 [Server] Running 32932ms behind, skipping 658 tick(s)
28.01 16:43:00 [Server] Running 4635ms behind, skipping 92 tick(s)
28.01 16:42:56 [Server] Running 19285ms behind, skipping 385 tick(s)
28.01 16:42:26 [Server] Running 10145ms behind, skipping 202 tick(s)
28.01 16:42:12 [Server] Running 14060ms behind, skipping 281 tick(s)
28.01 16:41:48 [Server] Running 9376ms behind, skipping 187 tick(s)

For 122 seconds (2440 ticks), the server skipped 1805 ticks, roughly 90 seconds. That means the server was running at about 5 tps during that time. Useless to say I cannot real-world-test anything under these conditions. What does it take for this issue to at least get a "confirmed" status?

@FVbico

He didn't complain about it not being assigned. He simply mentioned the still "unconfirmed" status. I agree it's a bit weird. Do they "deny" the issue?

No idea, but I do agree that the mods should update the confirmation status.

Oh no, I'm not complaining, just getting a little worried is all. To the best of my understanding, the bug won't be assigned until it's replicated and confirmed, right? Cause at this point, nobody can seem to pinpoint the exact cause. As for the theme, I know it's mostly a joke, but it seems to be a trend that the bugs fixed are related to the theme. Maybe I'm just misreading things.

@Early Reflections: It got so bad for me last week that the sun wouldn't even come up, and just trying to walk would trigger Dinnerbone's new anti-cheat system, and TP me backwards. I had just got done essentially refreshing the server map even more than usual. (I completely erased all command blocks and entities outside the shores of the main continent) and it responded by lagging worse than ever. O-o

Mojang does most assignments internally, you will not see that field populated until a fix is posted most likely. It does not have to be in a confirmed status to be worked, it just helps.

What would help far more is a known and reproducible cause. The world you provided, per your own description above unfortunately does not count as it has been heavily and repeatedly edited by 3rd party tools, thus Mojang has no clue what has been done. If anyone can identify the error in that world, and reproduce it in a new, untouched map, that would give the developers something to key in on. It is very possible, indeed likely there is more than one issue at hand here affecting different people. There are a ton of people NOT experiencing this issue (I've had no problems resembling this in any of my worlds) and that makes it very difficult to work (or even who to assign it to that would have knowledge of the necessary code areas).

In addition to all that, "lag" is way to general for them to try and do a targeted fix for and thus as @unknown said, they mostly get addressed at the end when developers can spend time on fishing expeditions.

Many people on this thread, have provided detailed traces of likely causes of this issue. The original reporter was not using a modified world. Almost all the people I know that run servers have stuck with 15w47c because the later snapshots are unplayable. My primary server suffers from this lag and was a new world with 15w51c. People that submit watchdog timeout crash dumps get them closed as duplicates of an unrelated issue because the "watch dog timer is working as intended", hence you see no server crash logs.

There is a LOT of evidence in this thread that points at new mob AI and path finding, is this the only cause? We don't know, but the closer we get to release without this issue being addressed, the less likely it is that we'll find out.

In summary, trading anecdote for anecdote, I don't know of any instance of a post 15w47c server that doesn't suffer from this problem, and I have direct experience of many such servers of various histories. For more evidence, look at Etho's latest MC video on youtube where he has unexplained severe lag in his single player world.

@Fienxjox thanks so much for your reply. I've been following this bug for a while, and very interested in seeing it through to resolution. Here is a link which I think narrows the problems down, at least somewhat. (http://www.mediafire.com/download/lz33a58qwmm0hut/SnapshotTest2.zip). The issue arises when a number of mobs must path-find (in this case a single long narrow path) and there are constant redstone (perhaps other types) updates.

As long as the burnout torch clock remains off the game behaves normally. If the clock is turned on the server load increases. I've also noticed, with the F3 screen their is an incredibly high number of allocations happening while the mobs are pathfinding.

While this might seem to be a contrived example, the same scenario might be found in a mob spawner. Having a large number of mobs preset with some redstone designed to kill the mobs. The same setup running in MC 1.8 produces no noticeable load on memory or cpu.

If the problem is not limited to redstone, and also happens with command blocks, then we might just have a cause and effect. I can conform that deleting BOTH all my command blocks AND mobs will alleviate the issue entirely, but neither individually will do so. It's also worth noting that as of the snapshots, I replaced most redstone clock operated command block contraptions with repeating command blocks, which would significantly increase the amount of updates to at least a once per tick timing.

@unknown we will take a look (I'm not on a computer I do MC stuff on), but to address your last paragraph - a contrived example is exactly what helps. If it can be reproduced in a brand new, empty superflat world with nothing else but the exact required setup that very much narrows down what could be causing the issue, and as a (non-Minecraft) developer that is always the best info a bug can have. Having it in a real/old world that has tons of stuff in it (Etho's comes to mind) makes it very hard, even if it is known where (roughly) in the world it happens because there is just so many possibilities.

@Jay: I just tested your provided world, and I can also confirm this working on my own system. I will do an experiment with MCEdit on my own world and see if that fixes things. Since most of my active contraptions have been converted to repeating command blocks instead of active clock circuits, this should pinpoint the issue without breaking anything else that might be causing it on my end.

My world has mobs and some redstone (no clocks but a chicken cooker that triggers intermittently) and suffers from lag.
I should note that the number of possible causes is SIGNIFICANTLY reduced by the fact that it was a change introduced post 15w47c, and hence a simple bisection should quickly identify the issue.

The most likely culprit is a change introduced in 15w49a, the "improved pathfinding". But 49a was so unstable it crashed all the time. 49b was released the day after to fix the crash issue. My guess is the issue wasn't really fixed, a band-aid was simply applied on the issue to prevent it from crashing. It's been running very badly since that "fix".

I think that's a good starting point to look at.

Well I just did some experimenting on my own world. Best I can tell, the problem in my case starts at 15w35e. Or in other words, the build where the "always active" function was introduced for command blocks. In 15w34d and prior, the lag does not occur. Furthermore, reintroducing the world into post 15w35e snapshots fails to reproduce the issue, but also resets all command blocks to the "needs redstone" state.

OKAY, I think I've found another possible culprit! Using Jay's test world, and a repeating command block set to always active to constantly update entitydata, causes the lag to resurface just the same as using the clock circuit. Curiously however, it ONLY happens in this case when a Zombie decides to recalculate it's path. They'll make a B-line for the Villager no problem, but if they decide to regroup and take the stairs, the lag will start.

It would make sense in my case that this was at least part of the issue, as I have at least two of these commands constantly making any Villager or item frame invulnerable, which on my server, accounts for at LEAST 500 entities.

In my case, the server ticks can fall down as low as 5 tps whenever there are mobs around. And my world doesn't have a single command block in it. However, it does have a simple repeater clock for an item elevator.

From the looks of things, that will also do it. I just tested on the same world, and essentially any form of redstone clock will cause the issue. In addition, while repeating command blocks do not appear to be a cause on their own, any command that pings a large area, or updates a large amount of data will trigger it. (Eg: testfor, entitydata) Curiously however, executing a simple command like /say from all entities doesn't have any effect, despite the nature of execute being just as broad as the other two.

Unfortunately, because of the way this bug seems to work, this means I can't actually test whether or not the old repeating command methods will work, (impulse block hooked up to a clock circuit) as I would not be able to tell whether the lag is caused by the command, the clock, or both. >->

There's a separate ticket for command blocks causing lag (#MC-94684), but I do think that THAT bug doesn't occur unless you have mobs spawned/spawning. In a test world, I seem to be able to make loads of repeating command blocks, as long as there are no mobs around. Once mob spawning is on, the lag starts and scales with the number of command blocks, and the number of mobs. Without the command blocks I don't see the lag (in a flat world with normal mob spawning). These two tickets should be linked at least, as they seem closely related.

Please recreate this in a new super flat would and link or attach that. If it can be isolated as much as possible, the devs can do the rest.

Here is the same world provided by Jay with my findings added to it. Simply use the zombie spawner with the button, and activate any of the redstone clocks in the area, or turn the repeating command block to the always active state to see the effect:

https://dl.dropboxusercontent.com/u/47112247/SnapshotTest2.zip

@fienxjox - are you actually in contact with the devs? I believe that there is plenty of evidence in the comments for the devs to find the issue, there's profiling output, information on when then issue started and information on how to re-create the issue (make a world, spawn mobs, introduce path finding complexity, watch world stop).
Simple experiments suggest that mobs in a super flat world do not cause issues, but it is relatively easy to trigger issues, simply introduce barriers to free travel and motivation for the mobs to path find.

I'm fairly convinced that the command block issue being experienced by Liam is a separate issue that is likely being severely exacerbated by the primary issue in this bug.

I also believe this issue is unrelated to MC-94684, as the command provided does not effect the test world on my end. To the extent of my current research, it only seems to be commands that target or effect entities that triggers it with command blocks. Though why /execute doesn't do it is beyond me...

@Pongo Saoiens the mods do have ways to contact Mojang directly but we very rarely contact them directly over a single bug report unless it is a security issue. We typically flag bugs behind the scenes for review which the devs do weekly usually. I will do that here, but beyond that the mods aren't involved in the process.

I can confirm this issue for 16w04a. Steps to reproduce:

1. Create a superflat world (“The Void” preset) in Creative mode

2. Set the time to night

3. Build a simple clock hooked up to a lamp or note block

4. Place a water source (far enough away from the clock to not destroy it)

5. Spawn some zombies next to the water

6. Spawn a villager on the other side of the water. You will notice the clock immediately slowing down, until the villager is killed or converted.

Could this please be added to the description?

I don't believe water has any part of it, as setting up a multi-path system to a villager on land will work all the same. The only thing water does is force mobs to constantly attempt to recalculate which path to take, which seems to be the heart of the problem.

I used water since I believe it to be the fastest way to set up the scenario.

Be that as it may, because of this, in other duplicates, people have come to the conclusion that liquids play some part in the issue. As far as I can tell, they do not.

We upgraded to 16w04a... 13 players online, eating takes 30 seconds or more, going through a portal takes several minutes, and other symptoms. We will have to revert to 15w47c again to keep the server playable. If there is some specific profiling Unknown2k can do, and then forward the data privately to Mojang, we players would be happy to "do our normal thing" and help zero in on a target for the problems that keep us bouncing back to 15w47c.

I myself experienced this for a long time. redstone ticks would take seconds, mobs would fall through the air slowly when hit. all the classic signs of tick lag. in desperation i made a backup of my survival world and did /kill @e and it did fix the issue. through more experimentation i've found that having lots of villagers in the world is related to the problem. Obviously, this needs to be fixed. Villagers are my favorite part of minecraft and I'd hate to not be able to make use of them anymore.

Yeah, I'm having this 'lag' problem too in 16w04a... Except my worlds are singleplayer flat worlds, and not open to LAN. I've made a few minigame worlds that use a bunch of the new command blocks (repeaters, chains, impulse) and little to no redstone. When played in 16w04a, my minigame worlds play fine for the first few minutes and then the world starts to lag, and it gets worse until the minigame is unplayable. But when played in 15w46a, my minigame worlds play flawlessly with no lag at all... Pretty strange stuff. Anyway, I'll play with this more to see if I can get a solid way to reproduce this problem.

I tried the test world that Liam Scott posted. The tests can be done with single zombies.

The lag only happens when the zombie has a choice of two or more paths. If you spawn a zombie away from the narrow path it will move to stand under the villager without causing lag. But if you spawn one near the end of the path you get lag. Same happens when one of the zombies under the villager decides to try use the path.

The one thing that stands out to me is that the game is not skipping ticks during this, and the game tries to quickly catch up at the end causing the redstone and zombies move at a high speed once the zombies reaches the end of the path. The faster the redstone clock is the worse the backlog of ticks gets. The repeater clock shows this without causing very long delays.

So one bug seems to with the way the AI handles multiple paths and the games inability to skip ticks that backlog during path finding.

@Liam I can have both java 8 and 9 installed on my pc w/o issues but on the server I had to uninstall all other java versions before installing java 9.

Unfortunately I don't have that option short of having my server host do it for me. Locally however, it made no difference for me, so I can't really be bothered asking.

If the game isn't skipping ticks, then this might actually be an unrelated bug from that which the rest of us are having. As far as I know, at least two users have posted logs indicating that the server, local or otherwise, is skipping a lot of ticks.

It does skip ticks at the end, just not during the path finding.

I am having a similar issue in snapshot 16w04a on my server, but it seems to be linked to redstone. When any kind of redstone is activated the server gets massively lagged, and when the redstone is deactivated, the lag goes away. It is so severe that even powering a piston freezes the server for a moment.

@Tom Grey

I think those Issues belong together, but maybe are not the same. The Pathfinding is one issue and the redstone-issue makes the pathfinding-issue more visible. If i turn one off (kill all mobs or deactivate all Redstone/Commandblocks) the lags stop.
Her ist my issue with Redstone: MC-94684

Confirmed in 16w05a. And a lot worse than before, by an order of magnitude.

Also confirmed in 16w05a... I have a singleplayer flat world set to 'peaceful' and no structures (no villagers/villages or baddie mobs), and there are some horses and pigs wandering around. If I start up a command block clock (digital clock face, uses /clone command a lot, no redstone), it starts to lag bad about 1.5-2 minutes in 16w05a. The same clock will run lag-free forever on 15w47c.

What's weird is, I have the same digital clock in a 'void' world (32x32 stone square and nothing else, biome is 'The Void'), with a few horses and pigs spawned in similar to my flat world. But in this world, the clock seems to perform the same whether I'm running 15w47c, 16w04a, 16w05a.

Unknown2k does not follow this thread closely, as he has his server to manage among other things. But if there is some specific settings we can use for profiling the current 16w* series snapshot, and then forward this info to a non-public email, I can poke Unknown2k to look here so the Devs can have data to work with. We have been running the updated snapshot every release week for several hours, then rolling it right back when the game becomes unplayable. As the release cycle nears completion, we are very interested in having this issue resolved on release.

Steps to guarantee generation of massive lag (effective in 16w05b):

1. Create new creative mode superflat world.
2. Build a container containing a villager that cannot escape but outside zombies can still see the villager.
3. /time set night
4. Spawn (with egg or /summon) 10 zombies about 20 blocks away from the villager. Zombies start tracking to and walking towards villager.
5. Before the zombies come within 5-10 blocks of the villager, pour water around the villager.
6. The game will begin lagging horribly to the point where a player in survival wouldn't be able to accomplish anything.
7. Continue spamming water for even more lag!

@Connor Johnston

Confirmed. You get the same results when placing lava too. You can also get close to a total jam when firing up a torch-burnout redstone clock. 16w05b

@Connor Johnston

It doesn't even have to be just zombies in my experience. Just try pushing 10-30 mobs around in water all at once, that will probably do the trick. (I've only tested this on villagers though, so don't quote me on this until we get more evidence.)

@Tom Grey

It doesn't surprise me that causes lag, too. I just tried to think of the simplest and easiest way to guarantee a reproduction of the lag.

Not sure if another "I'm seeing this too" is actually helpful, but...

I think I'm seeing this issue in the Ender Dragon fight in a world upgraded from 1.8. At first I thought it was a problem with my old unsupported video card related to the dragon fight itself, but it seems to work perfectly well until one or more endermen get mad at me, then performance drops from 10 or 20FPS to more like 10 or 20 seconds-per-frame, leading to pretty immediate death at the hands of the dragon or one of the endermen.

Which would make perfect sense if the lag is connected to pathfinding: endermen teleport away and then have to pathfind back around the pillars / portal / structures I've built. A connection to redstone would also fit, as I have a simple but very large redstone circuit in which a button at "ground" level activates TNT dispensers far above each pillar to blow up all the crystals.

I have these log in single player too:
{{{quote}
[11:31:35] [Server thread/WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 2092ms behind, skipping 41 tick(s)
[11:31:57] [Server thread/WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 8250ms behind, skipping 165 tick(s)
[11:32:12] [Server thread/WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 6955ms behind, skipping 139 tick(s)
[11:32:27] [Server thread/WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 4721ms behind, skipping 94 tick(s)
[11:34:25] [Server thread/WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 2233ms behind, skipping 44 tick(s)

}}

Would it be helpful to the Mojang bug-crushers if I attached a zip file of my clock world? In that singleplayer world, all you have to do is push a button to start the clock, then stand on a platform and watch the clock face. In 16w05b it will start to bog down after 20-30 seconds (on my computer). If it's run on 15w47c it will run indefinitely without any lag at all.

If I'm allowed to upload my world, should I upload it in the above 'attachments' section (I don't know if that's considered 'hijacking', that's not my intention), or should I create my own bug report (which would probably get crossed out as 'duplicate', and my world might not get seen by the devs)?

I found a very good example here. Check out Snocrash's XP & Gold farm, the one fixed for 1.9: https://www.youtube.com/watch?v=Nss_ix6yZqg

Download world file in the video description and open it in any snapshot past 15w47c. Use as described in the video. Enjoy the issue in its full glory! It jams so much that after a few seconds, the only way out is to force quit the game. It works perfectly fine in 15w47c.

This ticket relates to one of my issues: MC-96120

I've never noticed this issue in normal survival play, but the Snocrash farm does grind mob movement to a halt in 16w05b.

Still in 16w06a.

Check my ticket [MC-96803], there are simple testing world and videos with working 1.8.9 and not working snapshot. I really don't understand why this takes too long to fix. Game is really unplayable, everytime there is a mob which want to go to another place and path finding algorythm eats ours CPUs. And finally it crashed my server due to lag and watchdog kill it after 60 seconds. I have used vanilla and normal gaming over real game world. I didn't think that snapshot can have so serious issue. So sad...

How can this still be unconfirmed? Mojang you are more than welcome to use my server to observe how bad this crashes my system. Does not take long. The crashes do extensive damage - simple redstone clocks freeze, purple repeater always active command blocks sometimes need to be replaced to work again. This is really a mess. Been this way since the release after 15w47c.

It just means the moderators haven't updated the confirmation status yet. I've done so now.

I personally have experienced higher framerates in the more recent snapshots, but then again I'm playing on a mid-range average laptop.

I've fixed the first half of this issue and ProfMobius is going to add some more optimizations.

Still in 16w07a, but to a lot lesser degree.

Amazing progress has been made on this issue. It lags a lot less than in previous snapshots. But there's still a lot of skipped ticks when there's a reasonable amount of mobs around, and we can still feel the TPS going down, and see the choppy sky movement.

The SnoCrash gold farm example I posted above is a very good example of the issue in action. Try it in 15w47c and then in 16w07a to see the difference. On the other hand, the test world uploaded by Jay showing the issue now runs perfectly smooth.

Also confirmed it is still present in 16w07a but as @Early Reflections mentioned, it is to a lot lesser degree. But it is still not as good as pre-15w47c versions.

Third confirmation there. Though on my end, it's still present to such a minor extent that it's almost not even noticeable any further. Night and day difference.

I have problems loading terrain chunks because of this bug. When I press the Esc buddon and let chunks load, its fine till you try and load other chunks... Also, its a bit worse in F5 mode for me

@Liam Scott: make a redstone clock near any mob system and you'll see that there still is extreme lag.

@unknown Regular Survival-friendly redstone clock or do you mean old 1.8-CommandBlock-fillclocks?
Searge also commented here that it's just halfway done, I hope such things get fixed then as well }=)
We also still got the issue with Redstone Dust lag, but that bugpost is also already assigned, luckily.

Meri Diana I mean the regular redstone clock, no commandblock stuff.
Also I assumed that the other half was the optimizations that ProfMobius was working on and I assumed he was done with those since this was marked as resolved.

@unknown Not sure to be honest.. ;-;
But it's a good idea to collect inside this bugpost what still makes the game laggy/laggier (after 15w47c), so thanks for detailing the specifics of the clock more }=)

See, here's the thing that bugs me about this bug and this bugtracker in general. No matter how this bug is resolved, people are still going to say that "there's still lag... its not fixed" because of the awful title of this bug. It is too general.

That being said, doing my original test with a ton of mobs that try and path find through water has significantly improved. I could be really picky and say that its not fixed because there is still slight tick lag when I do the test, but the game no longer comes to a screeching halt when I do it.
Let's be honest, do we really expect path finding with a ton of mobs to cause absolutely zero lag whatsoever? While that would be great, i'm sure that any video game that is forced to do something like this (deal with A TON of entities and path finding calculations at the same time) would have some sort of a performance issue.

I still haven't seen this "bug" outside of a path finding situation on my own worlds, so I can't say that this bug really exists in 16w07a personally. I'm also not going to be picky and say that "omg theres still a litte bit lag with a ton of mobs pathing thrugh water. reopen pls!" If Mojang wants to optimize path finding more (and they should), then that's cool, but from the progress they made with this snapshot, I consider the path finding issue fixed.

Jono, I don't have any problems at all with any kind of lag when there is tons of mobs. A gold farm in the nether for example, gives me no lag at all as long as I turn off the redstone clock so I think that this is either a separate issue or its the same problem and the redstone clock is just pushing it over the edge.

@unknown 100% agree, it's a very generalized/diffuse title, lag can have tons of different reasons.
Maybe we should go into according bugposts that are more specified to a lag reason.
But thank you very much to your general positive statement:

Regarding redstone-related stuff go and comment here
I don't know other related bugposts, maybe someone could add them here, if they exist.

@Jono

I would tend to agree with what you're saying. 16w07a brings massive improvement over the previous snapshots, almost to the point of full useability.

Where I disagree is the issue being resolved. The most basic mob farm that uses redstone to collect the loot does still introduce noticeable server lag. I'm alone on my server (on a dedicated host) and I still notice it. So if it does it in my semi-basic setup, with only one person on the server, I can easily imagine the server slowing down to a crawl with 10 people on and 3 or 4 mob farms active.

When my server was running 1.8, there was no noticeable server lag with 12 people on and 3 active mob farms. Running my server on 15w47c is smooth as butter, with no lag at all. Every snapshot after 47c was unuseable.

In conclusion, the issue is definitely on the right path to being fixed, but in my opinion it's still unresolved and in an unreleasable state. I trust the devs to figure that one out!

@unknown Like I said, redstone-related lag could go here

@Meri Diana
I disagree, the bug title is very specific: a change (or changes) introduced between 15w47c and 15w49a (the next release) caused massive lag. This actually significantly limits the scope of what can cause it. The bug is resolved when performance returns to the same level as 15w47c (and it's pre-cursors), or the devs explain why that won't happen. As it stands, 16w07a is the first version since 15w47c that I feel I can realistically play on, but it's still far from the performance of that earlier version.

@unknown Here's the thing: While I agree to your reasoning, usually people who are desperate to get their lag fixed will insert their comments about it often in this bugpost, as it's got many votes and a quite generic title (except the snapshot mention), they often ignore that.
E.g.: There were mentions of lag caused by pathfinding mobs through water, but also mentions of redstone clock lag at mob farms.
If anyone could give details what exactly changed and caused the lag after 15w47c, and what of it got truly fixed in 16w07a, then people won't be as likely post their individual lag problems in here.

@Meri Diana

I agree. The problem is, I don't know if it's redstone-related or mob pathfinding-related. The mobs by themselves don't seem to cause any significant lag in 16w07a. Same for redstone, by itself I don't see anything bad. But the two combined makes the issue very noticeable.

Like I said before, and also like @Pongo Sapiens said, something very specific introduced in 16w49a is the cause of this issue. 49a didn't change any redstone behaviour but did bring significant pathfinding changes.

@unknown That's exactly what I meant why people ignore the snapshot mention:
They don't know what caused it, and, frankly, if someone notices lag, they usually won't test if the very same lag already occured in previous snapshots..
So they will post in here.
That's why it'd be great of anybody could tell us what exactly got changed, what was the issue, and how much of it got fixed now, so people won't insert their lag problem in here, if it doesn't match the "requirements" (the reason for the lag after 15w47c), and will rather open new bugposts or comment on the according bugposts for their specific lag problem.

@Meri Diana

In this case I totally agree. It would be nice to have some feedback from the devs about this issue, at least if it's AI-related or redstone-related, or both combined.

16w07a, gradually got worse on lag with only a dozen people, then did a solid crash. Attempts to restart wouldn't bring it back up. We reverted back to 15w47c.

Unknown2k can post a crash report to get some useful technical data. We would love to know what useful, specific data we can provide to get past 15w47c on our upgrade path. Again, 1) What debug setting/ profile mode would help get useful performance data before the game becomes unplayable, and 2) How can we get this information to Mojang so we don't publicly reveal player base locations and other private info from a public Vanilla Grief/ Raid server.

Well I offer my server for any testing Mojang wants. I am still running 15w47c but I have all the snapshots loaded on the server. I know with 16w07a the elytra no longer works. seems like every snapshot makes it more obvious this is a microsoft product. Everything past 15w47c will not run more than 3 hours w/out crashing and my server has 3 to 8 players almost 24/7. It has come to the point I am about to throw in the towel. Doesn't anyone a Mojang actually play the game? Or run a server? When it does crash weird things happen too - I had piston flip flops - actual stones would disappear. A simple redstone torch one repeater clock locks up. Purple repeat unconditional always active cb's randomly stop working, I copy the code, break and replace cb - reinsert code and it works again... Seems like you guys need to have a little conference and see what is going on.

@unknown [ ... ] you will need to make a jump mid-air to activate elytras (they will no longer auto-activate)
I can understand people are being grumpy if stuff gets changed (but hey, that's normal in snapshots) or don't work as they should (but hey, that's normal in snapshots), but please refrain from any snide remarks.

I know about the repeater-CBs problem, but can't remember the according bugpost or workaround for it, but it has been noticed.

By the way, Mojang got no obligation whatsoever to give us new stuff in a game where we paid only once.
They give their best to fix everything, but given the bad code heritage, it isn't easy to get the desired outcome, so please be chill }=)
I'd also like to link to a comment by a Mojira Mod which might make it better understandable for non-coders:
Please read this

In any case, don't expect a snapshot to run like a release version; that's what snapshots are for - testing, getting fedback, improving or fixing.

And, overall, please don't add to the stress the Devs and Mods already got; although I can, as I said, understand people being upset, it won't help (quite the opposite) if you don't neutrally state your problems, but load your emotional stress onto them to relieve yourself.
There are so many people doing that, and by now I'm more and more worried about how the Devs and Mods can cope with that.

I'd like to remind you all how stressful it was for Mr. Persson (Notch) to get hate and problems and what resulted likely in him selling Minecraft to Microsoft, so please don't do it with other people (in general, not only people related to Mojang).
They are doing what they can.

@Meri Diana
Please can we keep this bug on topic, despite the suggestions that it's some kind of dumping ground for random lag issues, it has mostly been very focussed. Meta-discussions don't belong here.

Thanks.

@unknown Aside from me also referring to the content of the post (e.g. adding Mr. Bergensten's tweet regarding the Elytra change), I know anything else doesn't belong here, but then also do tell those who make such remarks to keep at the topic.

This gets more and more over time, I observe the "mood" in the community since longer.

I'm having enough, I won't just stand beside doing nothing and see people getting bashed/bullied unjustifiedly, even if that turns the mods against me for meta-discussing and mute my posts. If you can't understand my reasoning here and find those remarks okay, then I can't get my point across you.

@Meri: Not seeing any bullying from a quick look. Please send a modmail on reddit if you wish to report a user.

@unknown Done.

So I tested this in 16w07b using the snocrash gold farm, which did better than 16w07a.

One part of it was fixed, in that pathfinding and restone ticks are no longer directly linked to each other. But the pathfinding of a large number of mobs still causes lag, but no where near as bad as before. And redstone ticks still backlog, causing the game to fastfoward for a bit to catch up.

To test this yourself download the mob farm from https://www.youtube.com/watch?v=Nss_ix6yZqg

Fy down to the repeater clock at the bottom and put a lever under the block with the redstone torch on the side to disable it. Fly back up to the trapdoor where you collect the xp and set your game mode to survival. Throw a snowball at a pigman and they will start pathfinding to you slowing the game down but not affecting fps. The lag will quickly clear when you change yourself to creative.

Next build a fast clock nearby, I used 4 torch burnout clock. With the clock running change to survival and agro the pigmen again. The game and the redstone clock will lag, But the lag is about the same as the test without the clock. Switch to creative mode and watch the game fast forward through a backlog of ticks caused by the redstone clock.

This test was done in Single player on my desktop. Window 10, (2) 2.8Ghz 6core Opteron 8439SE, 32Gb mem, Radeon HD 5870, Java 9ea(102)

@Pixie

I did the test like you said and I got the exact same results you described. I think this Snocrash farm is a very good reference point for this issue. Another test world posted here (by Jay I think but not sure, haven't found the post) demonstrated the issue very well too in a much simpler way.

This appears to be an issue again in 1.9-pre2. Easily reproducible using the Snowcrash farm noted above by Pixie.

Liquids, changing dimensions/loading chunks, redstone all cause framerate drop and latency even on SP in pre-2.

Still not fixed and I'd be disapointed if it's not fixed before the release of 1.9! Since its a huge unplayable bug?!

For me, the Ender Dragon fight was so lagged it was unplayable in earlier snapshots, mostly fine in 16w07a, but now it's worse again in -pre2. Didn't try it in any of the versions in between. I don't think it's quite as bad in -pre2 as it was before 07a. In -pre2 it's almost unplayable rather than completely unplayable as it was before.

(Again with the disclaimer, I am not staff, just a highly motivated and interested common player on this public server)

Unknown2k has been continuously running PreRelease 2 since 19 FEB 2016. We've had lag spikes, but we are dealing with it and they do not appear all that different from 15w47c performance. We also note Spigot plans to release the day after 1.9 next week, which we expect to further enhance performance. If we find we need to revert before or on the 1.9 release, I will note it here.

We do hope in the future that the Spigot "Timings" report may be built into Vanilla, along with some way for us to grasp and report problems such as caused Unknown2k to file the original report. We also hope there is some means built into bug reporting to upload such reports or logs without having them publicly viewable. It is a horrible thing to make visible player login locations on a public grief/ raid server. But we still want to support the Minecraft product with useful data to find and fix problems.

@unknown But that issue is most probably a server issue, reported by @unknown here MC-97445 (private ticket). This ticket here is more or less a client issue.

Kumasasa: How is this a client issue? Everything reported here has been in relation to servers.

Yes, nevermind.
I think I messed up something.

Here's a log from the gold farm in the nether.

Just like to point out that although the lag is server side, it still exists in SP worlds (just on the server side of the integrated server).

whatever they did in 1.9pre2 seems to have fixed it for me. Runs real stable now

@Devs:

Use Snocrash's farm as reference. Do your tests with it. When it runs as smoothly as in 15w47c, the issue will be solved. Currently, it's not. It's running extremely slow and it underlines the issue perfectly.

^ exactly ive been saying this since early January but they just put my issue in with this and said they fixed it, please help devs! Use snocrosh's farm and see for yourself

I can confirm, i have snowcrash zombie pigman farm made from scratch on my server and the moment i load it in, players start complaigning of lag to the point the server is unplayable

Does anyone have any concrete causes (sorry if they are hidden among earlier comments)? So far I have seen redstone and commands mentioned as causes, but loading chunks and switching dimensions also have this issue.

Still in 1.9-pre-release 3.

Just did a test on 1.9 pre-release 3 using the snocrash's zombie pigmen farm, in a world with nothing but the farm itself, and I confirm the issue persists. As soon as the zombie pigmen are aggravated, the game starts a slow paced framerate. Even the falling of the mobs to the bottom happens in slow motion. Turn difficulty to peaceful, and the lag is gone.

Btw, just to reasure did the same test in the same world on snapshot 15w47c. No lag at all.

Regarding Zombies: I did the same test with "difficulty peaceful" on our server (reported in MC-97860). The problem remains and the CPU is still at (or close to) 100%. Zombie AI may be an issue but it's not the only one. In one area where the problem arises we have a lot of the following: villagers, hoppers and free floating items (overfull guardian farm).

Maybe entity ticks are causing this lag?

Im running pre3 on a server with 20+ users, i have a large guardian farm setup as well as snowcrashs pigman farm and all run fine in SMP at the same time without causing any Lag issues

This is very much not fixed in pre3.
I don't understand why this hasn't been fixed yet, I mean it was introduced in the same update as the new pathfinding was it not ? How can this bug be so hard for them to track down and fix ?

They should find out if it really is the new pathfinding code that's at fault and if it is then exclude that code for 1.9 and reintroduce it later when it's been fixed.
I wouldn't be surprised if this isn't fixed for 1.9.0.

This is an actual bug that needs fixing before 1.9 is released, whoever is spending time fixing them damn boats and stuff need to get onto the real problems such as this, if 1.9 comes out and this shit isnt fixed there will be major problems (idc how long til 1.9 comes out but make sure this bug isnt present in it!! Its been over 2 months, how long do you need!)

Please do not respond here to this comment, the mods have enough work to do. If you'd like to respond, please do so on the Minecraft reddit (https://www.reddit.com/r/Minecraft/comments/47inh7/open_issues_after_19_is_released/).

But please also cut the devs some slack. I understand the general frustration, given how eagerly people have been anticipating 1.9. I am also dissatisfied with some of the prioritization. But code is hard. Code that has existed and been modified over a long period of time is harder. We have incomplete information as to the difficulties of addressing performance issues in general, to say nothing of this one in particular. I'm sure the developers are aware of the negative implications for the upcoming release.

Still in 1.9.Pre-release 4. I guess we can safely say "still in 1.9".

How bad is the lag on small servers? I'll need to hold off on updating mine if it's too bad.

@Roadsguy the lag appears in singleplayer on a rather powerful PC aswell so I doubt you can run any server with a even just a few people without noticing alot of lag unless none of the players are doing any redstone clock stuff or big mob systems.

@Roadsguy I haven´t seen any lags on my Server, and i have a lot of commandblocks running, some small mobfarms and everything is fine.

Please! This is not a discussion forum.
"How bad is the lag"
"Small servers"
"a rather powerful PC"
"doubt you can run any server"
"just a few people"
"Noticing a lot of lag"
"big mob systems"
These are all subjective statements and in no way contribute to the resolution of this issue.
Comments should be objective, factual, to the point, and relevant to solving the issue.

Thanks.

Still in 1.9, even the game chat will flood with Someting tooks tolong, root.tick, root and root.render
Even a server got the same spam.

True (but harsh) critic, this is not a discussion forum, but it's understandable that people are now anxious due to the release when such a big issue is still open };]

Constructive critic: Where can someone talk about that, if not here? > Solution:
If you want to discuss that topic with your individual experiences further, I'd suggest the Minecraft Forums, or maybe the Mojira Reddit.

^{Sorry again for metadiscussion, I just don't want anyone to feel "slapped" unfairly for trying to find an answer; everyone who seeks help should be welcomed and being helped, in one way or the other, and many don't know where to turn to.}

1. Can we please all stop being wanna-be mods and let the mods do their jobs? If you want to tell people that this isn't a discussion forum, become a mod yourself.

2. Another possible cause of this bug is the tick root.save. Sometimes it appears to stall the server when doing a save-all command or simple auto-saving. I'll be looking into this more today when I launch my 1.9 server.

Jay dude what are you going to do

@@unknown: Can you please stop bullying other users . If other users say "this is not a discussion forum" then for sure it's justified.
@All: STOP. This is not a discussion forum. For any meta discussion please use https://www.reddit.com/r/Minecraft

as extra proof of this lag, as if there wasn't enough:
my natural vanilla 1.8.9 average FPS: 16
my natural vanilla 1.9 average FPS: 2
my natural Forge 1.8-forge1.8-11.14.4.1577 (several mods) average FPS: 8

don't say anything about a bad computer

Happy to say that I've run my 1.9 server for a few hours now without any issues. It looks like root.save tick lag isn't because of this bug, so we can disregard that.

The Snocrash farm is the only way I have recreated this bug, where the lagging tick is "root.levels.world.tick.entities.regular"
"root.jobs" and "root" could also be a cause, as well as "root.levels.world.tick", and "root.levels.world.tick.chunkMap".

@mods can you please fix this problem? Cheers or give me a refund of the game, fuckwits

Unknown2k has been running 1.9 Vanilla since its release. The major problem has been DDOS attacks and the swarm of new players, some not following posted rules. While it has been laggy at times, it did have 83 players on it at one point. Unknown2k would have to post to say if it is resolved in his estimation compared to what we've been seeing, but we are running and it has been playable. We do plan to migrate to Spigot later this week.

Thank you for the support on this issue.

--RomaqRosher (Not staff, a just helpful player of the server)

@unknown, only developers are able to fix it, mods just manage the tracker.

The stuttering of monsters is less than it used to be, but the server's CPU usage still shoots up to 100% on 1.9. (2.8ghz CPU) when using Snocrash's farm. (We've built one from scratch, and it really does exemplify this problem quite well)

So is this a duplicate (or otherwise related to) MC-17630? I thought that was the bug tracking the seizing up of zombie pigmen in snocrash's farm (and zombies more generally) in 1.9.

@ziggurism

They may be related, but I don't think it's a duplicate. MC-17630 specify pathfinding to unreachable targets while this one concerns both reachable and unreachable targets. Of course this is if pathfinding is the definitive cause.

In Snocrash's farm, zombie pigmen see you as a reachable target, and they have multiple possible paths to reach you.

Confirming 1.9.1-pre1, 2 & 3.

I can confirm. If anyone thought these releases fixed issue. It did not. I am getting sick of all zombie mobs causing lag on even SMP servers. I have only 7 to 10 users and any mob relying on new tracking is throwing up errors. Please fix. I rather have stupid mobs than having server crash constantly trying to keep up with complicated player / mob tracking. Having Smart mobs is Totally not worth it.

Can confirm, too. I was testing out the XP and Gold Farm by Snocrash (https://www.youtube.com/watch?v=aBLEbJ9ozRA), it was really obvious there. Since 15w49a all the Zombie Pigmen walk very stutteringly, even up to 1.9.1-pre3. I have noticed that it has become a bit better, but the issue still is not solved. In 15w47c, any snapshot before and all of the 1.8 versions they were just walking smooth like butter.

Yep, it pretty well never left our server either, though I've yet to narrow it down to any specific cause. The snapshots and 1.9 releases have definitely LESSENED it, but I've yet to see a smoothly rolling sky pass over, and potions take ridiculously long to use. It's playable, but only just. Sometimes.

Currently, it gets a great number of "Can't keep up!" warnings, and skips anywhere between 100 and 2100 ticks depending on what we're doing. This results in Elytra being basically useless, and players moving around any faster than standard walking, (this includes just dropping from the sky...) will result in "Moved too quickly!" followed by a prompt teleportation back to the last location they were picked up by the server. I also constantly still get console spam about keeping entities with duplicate UUIDs, which is something that, at least on a redstone and command level, is know to cause issues, and I know for a fact that I never duplicated anything myself.

Quick snippet of info in case it helps: What I have observed by watching the memory usage is that the lag is accompanied by an unexplained ~40mb spike in memory usage.

When 1 player is on the server and the server is functioning properly, memory hovers around 70mb. At the same time the lag occurs, the memory usage of the server shoots up to about 110mb for no known reason. If there is a way to track what is using the memory, it may be possible to identify what is causing the issue (or at least something else that the issue is causing to use a bunch of memory).

On a side note, the lag and memory usage problem only occurs when a player is on the server. Without any players, memory hovers at 45mb and Avg tick at 0.25-0.5ms.

Also: Using /kill @e[type=!Player] doesn't stop the lag, nor does it reduce the memory spike. BUT if I do /kill @e and then click respawn, the server lag goes completely away and so does the memory spike. However, it doesn't happen when I just type /kill @e. If I fire /kill @e and I die, then the lag continues and will continue until I click respawn. The moment I finish respawning, the memory returns to normal and the accompanying lag ceases immediately.

I've discovered that this is because respawning changes the player location. In my server, going beyond about 510m in the x direction causes the memory and lag spike. I'll test a bit more off server to see if any lag happens there as well. I'll then test in a different world and using different entities and spawners and such to see what happens.

After testing, there are no mobs that cause any significant lag. When looking in the direction of the lag, I found out that the further and more often I pushed between the edges, then farther out the edge moved.

And so it seems that it is the terrain generation in certain locations that causes significant lag in the server. I have a hunch that it is more significant when this terrain generation is combined with trying to communicate the not yet generated chunks to the client, but that's just a hunch. Once the chunks have been generated, there is little to no lag.

I may attempt using a terrain pregeneration program to pregenerate all of the close by terrain and then see if the lag ever returns.

Hope this helps. I'll continue experimenting to see what I can find.

Every time you edit your post or comment, you are sending out an email to everyone who is also watching this issue. Just wanted to let you know.

I apologize. Is there a way to disable that?

Unfortunately not D:

You can preview your message with a blue button

I'm surprised I'm the first to mention this for 1.9 stable release...but then again, I hadn't been playing MC for a while and wasn't there for the snapshots mentioned, so maybe it's not as bad now as it was before, but our server (running 1.9 stable) still has this issue.

I recently made snocrash's pigman grinder, and if I go up and activate it, the following WILL happen:
1. Lag will increase noticeably for all connected players
2. The pigmen will start jittering as they move (due to server taking many seconds to process all the pigmens' AI)
3. The server will crash
These happen consistently, every time.

We've noticed lag at other times as well, but the above example is the single most extreme case, and proves there is still a major issue with the AI (and yes, the pathfinding does seem most likely, as others have mentioned, since the pigmen in the grinder can't actually reach you and it will push the pathfinding algorithm to its limit, whatever that limit is).

Steven mentioned a sudden jump in memory usage. This in combination with other evidence makes me think it's probably an issue with method recursion. This is just the most common culprit for similar situations I've seen before. In many cases, each level of recursion must retain any temporary variables / data structures / objects / etc it has created until (at least slightly after) the final recursion returns, so if you're not careful, it will easily use up all resources available to it almost instantly (unless there is some kind of delay in the method...or a few other exceptions).

I'm guessing one reason they're not just moving back to the old algorithm is because some of the new features rely on the new one...?

TLDR This still seems to be occurring in 1.9 stable release SMP vanilla; can any confirm?

@unknown

1. Lag will increase noticeably for all connected players
2. The pigmen will start jittering as they move (due to server taking many seconds to process all the pigmens' AI)
3. The server will crash

This are also the symptoms of an out of memory issue, please assign more memory to Minecraft and retry.

I can confirm what kyle has reported an in terms of memory, my box has 32gb RAM an it still isnt enough when you use snocrash pigman farm

"I'm surprised I'm the first to mention this for 1.9 stable release"
My comment was about the 1.9 stable release. It is also a problem in the 1.9.1 pre-3 that I tested.

However, when it comes to the usage of resources, the lag comes well before any memory is full. I play with 2GB memory or the server, but it was only using 130MB and the lag was still there. It is processing resources that are getting used up by what appears to be an extremely inefficient algorithm in terrain generation. If there is an inefficient algorithm in pigman pathfinding, I have not encountered it myself.

TLDR - Some functions are working way too hard to accomplish certain tasks and they are causing lag.

I can confirm it too for 1.9 release and 1.9.1-pre3.

Taking Snocrash's farm as reference, no matter how much memory I assign to the client (I tried 8GB), it doesn't matter a bit. Everything slows down to a crawl.

I'd tend to agree with Kyle Koder (above) about the pathfinding being pushed above its limits, gobbling up every resources available. Perhaps the routines create and delete too many objects/variables through recursion, pushing a big demand over the garbage collector.

I'm too running a server on 1.9 and I'm having real trouble with keeping it stable. We've built a zombie afk farm out of a spawner.
In previous versions of MC it was possible to afk and have a server still running at around 700 to 1000 zombies. My server can be repeatedly crashed now at around 300 to 400 zombies spawned. The server spams these messages:
[Server thread/WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 10630ms behind, skipping 212 tick(s)
And when the zombies reach critical mass the server declares itself as crashed and shuts down.

But memory isn't the problem. The server runs on debian and got 8GB RAM assigned in the java arguments. So I watched it happen while having the server output and "top" in two sessions open. The server starts spamming the "skipping ticks" messages at around 230 to 240 total entities, with a majority of zombies. The memory usage doesn't really go up. The problem is the single thread CPU load. Every zombies seems to increase the load on the one core that the server uses by about 0.3 to 0.4%. Once it reaches 100%, the server starts spamming the "skipping ticks" messages. I guess it's because the system tries to use more than one core for the server at that time to handle the above 100% load. The load goes down on one core to about 50% and another core load starts to rise with about 45% in the beginning, rising for every new zombie. I guess the minecraft server can't synchronize the load balancing that's happening here, so it will shut down eventually.

Look through older comments... this has been confirmed for 1.9 release. Several times. Anyway, I think the issues are in pathfinding and chunk generation. Profiling of the issue earlier in the comments found the performance increase was in the "other" category, I think.

Here is an analysis video by Slicedlime: youtu.be/m8hLFol9j5M
In short: It seems like the amount of calculation goes up quadraticly with the amount of entities, instead of linearly, like before.

This issue is duplicated by/relates to MC-98822.

This issue is specifically for servers, and does not necessarily have to do with the entities as terrain generation is also playing a part in this lag. Each issue should probably be separated into their own bugs, but this bug just says "server lag" with no specifically named cause, so the two known causes end up mentioned here: Terrain generation and entity path finding. I'm unsure what the protocol is for such a situation.

I'm running a public 1.9 server, and am also noticing the issue. I've noticed that the percentage of the allocated RAM continues to gradually climb, as though something's being cached somewhere. I've set the server to restart once an hour, which seems to clear out whatever's being cached. Some players are noticing issues with block lag around the 30 - 45 minute mark after restarts, which continues to get worse until the next scheduled restart of the server.

Sometimes this happens in singleplayer, but not as often, this game output is from March 12, 2016:

[18:47:33] [Client thread/INFO]: Setting user: Foltrox
[18:47:36] [Client thread/INFO]: LWJGL Version: 2.9.4
[18:47:38] [Client thread/INFO]: Reloading ResourceManager: Default
[18:47:39] [Sound Library Loader/INFO]: Starting up SoundSystem...
[18:47:40] [Thread-5/INFO]: Initializing LWJGL OpenAL
[18:47:40] [Thread-5/INFO]: (The LWJGL binding of OpenAL.  For more information, see http://www.lwjgl.org)
[18:47:40] [Thread-5/INFO]: OpenAL initialized.
[18:47:40] [Sound Library Loader/INFO]: Sound engine started
[18:47:42] [Client thread/INFO]: Created: 1024x512 textures-atlas
[18:47:58] [Server thread/INFO]: Starting integrated minecraft server version 1.9.1-pre3
[18:47:58] [Server thread/INFO]: Generating keypair
[18:47:59] [Server thread/INFO]: Preparing start region for level 0
[18:48:00] [Server thread/INFO]: Preparing spawn area: 68%
[18:48:00] [Server thread/INFO]: Changing view distance to 4, from 10
[18:48:01] [Server thread/INFO]: Foltrox[local:E:6fb6a471] logged in with entity id 0 at (-207.70037569986047, 8.162496554047785, 2148.5317391671583)
[18:48:01] [Server thread/INFO]: Foltrox joined the game
[18:48:02] [Server thread/INFO]: Saving and pausing game...
[18:48:02] [Server thread/INFO]: Saving chunks for level 'Kill Villager'/Overworld
[18:48:02] [Server thread/INFO]: Saving chunks for level 'Kill Villager'/Nether
[18:48:02] [Server thread/INFO]: Saving chunks for level 'Kill Villager'/The End

(Lag Begins Here)

[18:48:05] [Server thread/WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 2658ms behind, skipping 53 tick(s)
[18:50:22] [Server thread/WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 92794ms behind, skipping 1855 tick(s)
[18:50:36] [Client thread/INFO]: [CHAT] [Debug]: Hitboxes: shown
[18:56:38] [Server thread/WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 2386ms behind, skipping 47 tick(s)
[18:57:02] [Server thread/WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 2174ms behind, skipping 43 tick(s)

(Lag Stops Here)

[18:57:19] [Client thread/INFO]: [CHAT] Saved screenshot as 2016-03-12_18.57.19.png
[18:58:28] [Server thread/INFO]: Saving and pausing game...
[18:58:28] [Server thread/INFO]: Saving chunks for level 'Kill Villager'/Overworld
[18:58:28] [Server thread/INFO]: Saving chunks for level 'Kill Villager'/Nether
[18:58:28] [Server thread/INFO]: Saving chunks for level 'Kill Villager'/The End
[18:58:29] [Server thread/INFO]: Saving and pausing game...
[18:58:29] [Server thread/INFO]: Saving chunks for level 'Kill Villager'/Overworld
[18:58:29] [Server thread/INFO]: Saving chunks for level 'Kill Villager'/Nether
[18:58:29] [Server thread/INFO]: Saving chunks for level 'Kill Villager'/The End
[19:07:45] [Server thread/WARN]: Skipping Entity with id Minecart
[19:07:45] [Client thread/INFO]: [CHAT] Unable to summon object
[19:08:54] [Client thread/INFO]: [CHAT] Saved screenshot as 2016-03-12_19.08.53.png
[19:09:01] [Client thread/INFO]: [CHAT] Saved screenshot as 2016-03-12_19.09.00.png
[19:31:33] [Client thread/INFO]: [CHAT] Saved screenshot as 2016-03-12_19.31.32.png
[19:37:39] [Client thread/INFO]: [CHAT] Saved screenshot as 2016-03-12_19.37.38.png
[19:40:35] [Client thread/INFO]: [CHAT] Saved screenshot as 2016-03-12_19.40.34.png
[19:46:29] [Server thread/INFO]: Saving and pausing game...
[19:46:29] [Server thread/INFO]: Saving chunks for level 'Kill Villager'/Overworld
[19:46:29] [Server thread/INFO]: Saving chunks for level 'Kill Villager'/Nether
[19:46:29] [Server thread/INFO]: Saving chunks for level 'Kill Villager'/The End
[19:46:32] [Server thread/INFO]: Stopping server
[19:46:32] [Server thread/INFO]: Saving players
[19:46:32] [Server thread/INFO]: Saving worlds
[19:46:32] [Server thread/INFO]: Saving chunks for level 'Kill Villager'/Overworld
[19:46:32] [Server thread/INFO]: Saving chunks for level 'Kill Villager'/Nether
[19:46:32] [Server thread/INFO]: Saving chunks for level 'Kill Villager'/The End
[19:46:35] [Client thread/INFO]: Stopping!
[19:46:35] [Client thread/INFO]: SoundSystem shutting down...
[19:46:35] [Client thread/WARN]: Author: Paul Lamb, www.paulscode.com

Please can we get an updated fix version. The current one is in the past. This is a major issue for Vanilla servers and it would help to be able to give them some line of sight to resolution. Thx.

Just curious - are you saying 47b is fine and 47c is the first bad version, or...?

@unknown

after 15w47c have been unplayable

It mentions it in the description, and you can look in the affected versions. 🙂

Confirmed in 1.9.1.

Also still present in 1.9.2.

If anything what ever they've done has increased load on servers since 1.9 ...running a server its getting to be a joke for resources and cpu esp with 15+ users

Without heavy lag control plugins, vanilla SMP is unplayable. Why hasn't this been properly fixed yet?

I'm not sure if anyone has mentioned this yet, but the lag is also an issue in single player, it's basically made both gamemodes unplayable! I, along with several other people, would greatly appreciate a fix for this! Or at least some news as to what might be causing this and what's been done (if anything) to fix it?

I decided to try playing Minecraft again after a while away. Got updated, tried single player and wow, is it ever bad. It's literally unplayable. I just want to hit trees and get wood, but I have to wait several seconds for each block I break. This release definitely should have waited... Definitely should've been tested more. This is the worst performance I have ever seen from this game.

Playing on Arch Linux with 4 GB allocated for the game.

Could this be related to MC-100382 at all?

That would probably be the cause one of the issues, but NoAI entities (no pathfinding) still cause lag and so does loading chunks.

My family's Realm has been almost unplayable following the release of 1.9. Machines are breaking down, rails are unusable, potions/food take five tries or more to actually take effect, ender pearls can take 15-20 seconds to activate when thrown. This is abysmal.

In past versions my hardware has handled the 32 chunk loading distance just fine. But I think that's a major part of the lag. I lowered to 24 and the lag is almost completely gone. So it's a good workaround for now, try lowering the view distance. Hopefully it gets fixed soon so I can get my far view distance working again.

I think this has to do with Grass blocks triggering Chunk Loads for Light Level checks. As the profiler report in this report shows high tickPending.

As that is another flaw we've recently found and fixed in Paper (Spigot) Server.

My quick bandaid for that in paper is to return 0 for unloaded chunks light level... Which should only ever happen for blocks that gather light data from neighbor, and that would result in it using 1 of the other 3-4 neighbors.

Maybe that's good enough for Vanilla to do?

There were no grass related changes in two months either side of the mentioned 15w47c.

@unknown It's not a change in grass itself, but grass blocks can choose a random block that's on the edge of an unloaded chunk. Then that calls getLightLevel, and the light level check will check its neighboring blocks light level to calculate what it should be.

The big difference in 1.8 vs 1.9 here, is that pre 1.9, there was a boolean in ChunkProviderServer that controlled whether or not stuff like this could trigger chunk loads. It was located between the Chunk Loader and the Chunk hashmap

That boolean was removed in 1.9, so anything acting on unloaded chunks, triggers a chunk load, where as in 1.8 and before, it was limited and did not always load chunks.

Maybe bringing back the empty chunk thing from 1.8 will help solve many of these issues.

That boolean was removed 10 weeks before this ticket, and additionally was always set to true in Vanilla.

Can anyone reproduce in 16w14a?

It's way better in 16w14a for me, but I don't know if it's the same cause as this one.

Hi @unknown, I'm sorry, but it's not yet resolved. As long as it is actually related to my bug MC-94570, it is not. I've just tried it out last weekend with 1.9.2 and still instantly 100% unplayable in single player.

I resolved it as awaiting response, due to the question asked earlier: "can anybody confirm for 16w14a".

I'm confused, because MC-94554 (Zombie Pigmen Anger = Lag+Server Crash) is marked as a duplicate of this bug. But another bug MC-17630 (Zombie pathfinding to unreachable targets causes server lag), which appears to be the same issue, is not. For what it's worth, MC-17630 is definitely not fixed in 16w14a: zombie pigmans are freezing in midair still. If that is somehow a duplicate of this bug, that one should be closed and this one reopened, but I may be confused as to how or whether these three bugs are related.

The pathfinding bug (MC-100382) has/should be fixed in 16w14a, @unknown.

So, is this still an issue in 16w14a?

@unknown My MC-100382 has nothing to directly do with performance issues of the pathfinding system itself. and would not directly relate to MC-17630

That bug fixed in 16w14a is a slow leak, that's been in Minecraft for years, and plenty of servers run fine for hours/days on end.

So this is still in 16w14a?

The description of this bug is too vague for me to test, but MC-94554 which duplicates this bug (according to @unknown) is easily verified to be still in 16w14a.

There were two things causing lag. We never specified which if them is the focus of this bug so this bug report took the reaponsibility of both. Those two thigs are 1- Pathfinding issues and 2- Terrain generation.

So until both of those (and perhaps other undiscovered causes) are fixed, it would be best to keep this bug report open, or to split up each cause into its own bug report and move all the votes and watches from this one to those.

Please note that the only person that attached terrain generation to this bug was Steve Mortensen, I have not noticed any change in the behavior of terrain generation associated with the transition from 15w47c to 15w49a, which is what this bug report is explicitly about.
It is not about generalized server lag, it is about a huge dip in performance that started in 15w49a, and is directly related to pathfinding (which was also one of the major changes introduced in 15w49a). There is detailed analysis that demonstrates this in the comment chain.

It seems entirely possible that this bug could exacerbate the memory leak described in MC-100382 but given the age of the underlying problem in that bug, it seems unlikely to be the root cause.

Not fixed for me in 16w14a.

I'm starting to think that this isn't just 1 issue but it's multiple issues each adding to the lag. Anyway, its better than what it was before (for me), now it only takes ~5-10 seconds for the zombies to speed up and when they're moving slow they aren't quite as slow as they used to be.

I had it with mass chunk loading/unloading (e.g. going through a portal); that's not quite terrain generation but similar.

MC-94554 appears to be resolved by this update. Use of example world 1.9Fix-Ultimate-XP-Gold-Farm-bySnocrash no longer causing server to crash with the snapshot 16w14a.

The Zombies do move slowly while trying to track on you. When they figure out their paths you experience a strange speed up effect as they charge you. Then next batch spawns in and the process repeats. FPS remained at near 60 entire test.

Not sure how that relates to MC-94438.

I was actually voting for and following MC-94554 but it was combined in this bug report.

@unknown If this is pathfinding related, here is another patch of mine I've been using in production just fine for a few months now:
https://github.com/PaperMC/Paper/blob/44a1d43781efe9792299457685731847b1a93e92/Spigot-Server-Patches/0061-Optimize-Pathfinding.patch

Drastically helped with failed pathfinding.

Ok, now I can definitely say that 16w14a lags a ton, too. First it was so good that I was able to set my render distance higher than in 1.8, then it got gradually worse and now I get single digit fps with minimum graphic settings and 2 chunks render distance. Seems like it gets worse and worse the longer I play, indepent of what world I'm in. Closing and re-opening the game doesn't help, also kill @e[type=] with various types of monsters and Item didn't help either.

Considering there is still lag despite the pathfinding being fixed, what is the cause of the lag when it comes to this bug? Should this bug be closed while there are still massive lag from any cause?

For me, no matter how many of whatever creature there is in the world, they don't create lag for me, however loading chunks creates tons of lag. Not even necessarily generating the chunks, but just the server trying to send the data to the client creates intense amounts of lag on the server unrelated to memory issues. Can anyone else confirm this issue for me? I've gotten it on every 1.9 server I've run.

It's also not chunk loading for me, I stood completely still.

I think the title should be changed to "Server: Massive Lag in any versions post-15w47c" since it doesn't only affect snapshots.

Updated, but nb: we are still unaware of the actual version - no one has stated 15w47b etc is fine.

I see a redstone-lag bugpost (imo falsely) duplicating this bugpost here: ^{Fixed/changed now by Kumasasa, thank you }=)}
In case any of you gets lag due to redstone contraptions: That might be MC-81098

We noticed a huge lag/lag spike problem with simple dust-piston-lever contraptions: There are maybe 10 redstone updates; everytime we switch the lever this causes the game to freeze, we thought possibly also due to lighting updates, but even encasing the pistons and everything and it being either in a void world or flatworld with no mobspawns didn't help.
We tested in 1.8 and 1.9, it already happens in 1.8, it's gotten worse in 1.9 though, and our hardware is really decent.

I'm just saying this here for those who might not know about the other post and might experience lag due to their redstone machinery.

It seems to me that there are different issues who can all lead to lag, and it isn't easy to separate all issues from each other, that's why it'd be better if we could separate them into either single bugposts or make some sort of table where we can refer to each of them separately, or it gets confusing, as this seems to be now some sort of "collection thread" (mobs pathfinding, mob amount, chunk loading etc.pp.) for anything regarding lag, and thus it might be more complicated to fix it, as we might be talking about different issues.

@Meri Diana I just asked, MC-81098 has nothing to do with changes from 1.8 to 1.9.

@unknown I'm about to upload a performance test video in MC-81098 with a more than decent hardware prerequisite.
As I said, it can be light updates issues, but redstone dust lag does for sure add to the problem.
And there IS a change between 1.8 and 1.9, as for the same mechanism (which, again, lags/lagspikes ALSO due to redstone dust, but that's NOT the main issue, it seems).

@unknown We are aware of the actual version, several people have stated that all releases 15w47c and earlier did not suffer from this bug. This bug report is EXPLICITLY about lag that started in 15w49a. If you are looking at lag that existed before 15w49a, then it is NOT this bug.
@unknown once typo-ed 15w49a as 15w47a, but apart from that it's been consistent.
So it is back at the head of the comments, the lag we are discussing here WAS NOT PRESENT in 15w47c, it STARTED in 15w49a

There is also big lag issues with collisions in 1.9. When several entites collide in a 1*1 spot, it generates a lots of lags. Ref : MC-100692

Knowing the fact that it started with 15w49a and not 15w47c, I looked at the commits for this specific snapshot.

Right now, I can't see anything that would justify the amount of lag reported and I couldn't reproduce the reported lag either, even with a large amount of mobs present.

Anyone with the problem, can you post an affected map ? This would help a lot if I could reproduce the lag.

@unknown I will check your patch on Monday, but without a reproducible case of massive lag, I can't say if the problem is solved or not.

>Anyone with the problem, can you post an affected map?
For me it's on any and all maps when chunks are being loaded. There is no specific map that has the issue, for me.

Sometimes the server chokes on the chunks and will stop all other processes just to load chunks causing the Avg tick to shoot up to 200+ms causing a lot of the "server can't keep up" errors. I could have thousands of mobs and there won't be any issues. I can run tons of command blocks no problem. But the moment I move my client causing it to ask for chunks, the server chokes up massively and seems to dedicate the entirety of its processing to the task while at the same time not being able to accomplish the task. Amplified terrain is especially bad at this, jumping the Avg tick up to 600+ms.

I have an odd feeling this may be related to the chunk loading/rendering bug in the client, but I'm not sure.

TLDR - The only thing I've found that consistently causes unreasonable lag is chunk loading.

@unknown, I think what you're seeing is MC-98994.

Possibly. I find this note really interesting: "In 15w37a and subsequently 1.9, this method was removed."

That may very well be exactly what is causing the problems. If that gets patched and fixes this problem, that would be fantastic.

@ProfMobius: Isn't Snocrash's gold farm a reliable world to show this issue?

F3's pie chart is great for telling you what is going on with client side lag, or "everything as a whole" in single player. It is a pity an /op can't get the server's perception of lag the same way. Or something of the data provided in Spigot /timings.

You can use /debug start and /debug stop on servers as an Operator to get similar log files to explain what is going on on a server.

ProfMobius are you unable to reproduce the lag even with the nether zombie farm ? Sorry if you answered this already somewhere but I don't have time to check.

I tested this again in 16w14a using the snowcrash gold farm. It can be downloaded from https://www.youtube.com/watch?v=Nss_ix6yZqg

The lag is still there not much different from the last test I did after Searge's fix.

This time I added trap doors from the middle ring to the platform in the middle to see what changing pathing finding does, and the amount of lag reduced. And lag seems to be the lowest when memory usage is low and increases as the memory usage goes up. So I changed the memory from 8 Gb to 3 Gb and that seemed to smooth the variations in lag. I also knocked the bottom out of the gold farm and put lava below to get rid of items.

Since more paths to the player reduced the lag, it would seem the lag is being caused by mob collision or a longer path search radius to the player.

The extra paths dropped the lag by quite a bit.. Will add a image of the trapdoors.

Found the issue.

Performances should be back to normal in the next snapshot.

Yay!

Yay indeed 😸 thank you!
</meta>

@unknown PS: Would you mind sharing what it was? Or rather: What it fixed?
As there are many performance issues/lag reasons, it'd be probably good for the community to know which TYPE of lag was existent since 15w49a, thus which type of lag(s) your fix might hopefully resolve.

Because I have/had the feeling we were talking about different issues at some point, so separating each lag issues from each other would be great, if we knew.

I hope it isn't too much asked of you, I know you guys are super busy and have better to do than replying to everyone who asks you to do so, but I'm sure the community is curious }=)

Please don't use this bug tracker as a discussion forum.

How is that comment "like a discussion forum"? In my opinion knowing exactly what performance issue was fixed is quite helpful in actually being able to test it: I agree that this ticket has ended up covering many issues.

^

Entity lag I detailed here: https://www.youtube.com/watch?v=m8hLFol9j5M is still exactly the same in 16w15a, much lower performance than the same test in 1.8.9.

@slicedlime I think it's a different bug, MC-98822.

Yes, possibly. I didn't know about that bug.

The bug described in this ticket was due to a pathfinding error. It can be tested via snocrash gold farm map.

It was also applying to any location with a large vertical empty space with mobs on top of it.

Any other lag related discussions should be kept to their respective tickets.

That makes sense actually. On my map, we have a lot of Villagers permanently, essentially trapped in their respective business locations, and below those areas often exist large open caverns, (man made, essentially underground cities) with more empty space than anything else. We essentially have villages on top of villages, and the only thing that makes the lower ones work at all is precisely placed skylights to the surface. In hindsight, I don't doubt something would eventually go wrong with that setup.

My personal experience: After the last try to fix it (previous snapshot) the lag got worse and worse the longer I play. Now in the new snapshot it seems like the permanent lag got replaced with lag spikes. It's not as bad as before, I often get 60 fps on my laptop with decent settings, but sometimes there's a short, but bad lag spike, which is especially bad in fighting situations.

Oh, I thought the lag in this ticket was described as lag in general. In that case, can we get someone assigned to MC-98994? Because my server is unplayable until that's fixed. Thanks.

If fixed, would this resolve singleplayer and menu lag?

Tried latest snapshot, menu still lags. I'll also mention that VBO's, Vsync, minimum render distance, OptiFine, allocating RAM don't work. Do not mention getting new/upgrading PC, graphics card, drivers, etc. as it is not possible.

@unknown: See MC-90265

Well mob pathfinding was the first discovered cause of 1.9 lag so it is logical that this ticket covers that.

Kumasasa I don't have that issue. My issue is that I'd always lag, no matter what (even if I like, allocate RAM, use VBO's, turn down render distance, etc.). For some reason, even all the menu would lag.

@unknown: That all has nothing to do with this ticket.