[MCPE-68796] Hoppers pushing full shulker boxes into a shulker box or items into full chests causes server lag

To repro, Fill 5 shulkers full of items, place those five shulkers into a hopper. Face that hopper into an empty shulker. Next steps prove that it isn't client side

Large amounts of lag will occur. The easiest way is to go to another dimension or an area a long way away (as a way to show it is server/host side not client side), and place gravity blocks while the hopper with 5 full shulkers are loaded on a second players device. This will cause entity rubber banding and block placement lag server wide.

Empty or partially full shulkers seem to cause less lag, this leads to the assumption that the data in the shulker is a contributing factor to the lag. Based upon this, the assumption is that the data in the shulker is put a transfer request then canceled ever tick,

https://youtu.be/eTPg_hXs8hM world file is attached.

A gametest was created to directly show the measurable lag. see BedrockTechnicalTestTools.mcaddon

lag:shulker_lag is the test for bring the shulkers in

lag:check_lag uses the time function in java script to measure the time between ticks...

Linked issues

is duplicated by 2

MCPE-136154 Optimize the hopper full check function Hopper::_isFullContainer Resolved

MCPE-137940 Hoppers cause immense lag when attempting to place blocks/items into the chest with no open slots Resolved

relates to 1

MCPE-27167 Shulker boxes filled with items cause incredible lag Reopened

Attachments

BrokenRealm.mcworld

TechnicalTestingFrameworkBP.mcaddon

TechnicalTestingFrameworkRP.mcpack

Comments 18

GoldenHelmet 2020-04-18T02:51:58Z

I was unable to reproduce this in a small test world (on an iPhone 8 running iOS 13.4.1). I filled 5 shulker boxes with cobblestone and put them into a hopper pointed into another shulker box. Animals and falling sand appeared to move at normal speeds.

Is it possible that the lag you experienced is, instead, the accumulation of multiple resource-demanding events taking place in your world, and the shulker-hopper setup you made was just the last straw? Would you be able to provide a video or test world?

ravinmaddhatter 2020-04-26T17:44:08Z

did you try on a realm or server? 20 shulkers in this config will cause item transfer rate to reduce to 1 item every 4 seconds? I did put server in the title.

ravinmaddhatter 2020-04-26T18:05:52Z

I have uploaded the world, upload the attached world file to a realm or server. i will upload a repro video to youtube in a few minutes and attached the link.

ravinmaddhatter 2020-04-26T18:12:18Z

https://youtu.be/eTPg_hXs8hM

This video explains how easy this is to reproduce. I didnt put it in the BDS or Realms pool because it impacts both. a Jira Admin can move it if they choose.

GoldenHelmet 2020-04-26T22:11:29Z

Thank you for the additional information. I didn't realize you mean BDS/Realm because the local game also has client and server sides and their interaction plays a role in some bugs.

8 more comments

ravinmaddhatter 2020-04-27T01:43:39Z

I was also able to reproduce in a single player world, it just takes far more hoppers. with 30 hoppers, my computer is heavilly bogged down. Realms are just really really weak and susceptible to lag.

The faster your computer is the more shulkers pushed into hoppers are required.

GoldenHelmet 2020-04-27T11:23:16Z

Thanks again for the update. Confirmed in 1.14.60. On my iPhone 8 I started to notice slight lag with 8 hoppers trying to push full Shulkers, and by 14 it was extreme.

ravinmaddhatter 2020-05-07T17:16:39Z

One note:

After playing around with this a fair bit, i believe the important data in this ticket is as follows:

Full shulker's have more impact than empty shulker
lag is reduced when the hopper target is full
CPU read write speed seems to impact repro (IE realms and low end servers are more susceptible)

From that information, i believe this implies the server is copying the data for the shulker transfer, including all of its stored data, before checking the container type. However it does appear to check the container "fullness" before attempting the push. If the container type was checked at the same time the container fullness the data transfer would not occur and the lag would be reduced.

This may an over simplification of the algorithm, however it from testing that seems to be what is happening.

If it stays in, i know this can be exploited as a lag switch, it can be used to do things like obtain objects intended to be unattainable, duplication, and other game breaking exploits.

ravinmaddhatter 2020-11-06T01:32:19Z

Note: seems to be reproducible with the equivalent number of hoppers pointing into full chests as items in shulker boxes. This may seem confusing but here is the scenario

54 hoppers facing into 54 full chests all full of items, you can get equivalent lag to 4 shulkers on a server/realm. The lag caused is way more several people being online. it does bring our server to it knees.

Gustavo Dominguez 2021-04-09T18:23:24Z

this bug is considerably worse than we thought, or at least it has gotten really bad in .210 for some reason.
my server specs: Ryzen 2700@4ghz, 16 gigs of ram@3000cl18, running Trapdoor for profiling, Windows BDS for .210, locally, max threads 0 (or as many as it can use).
for this test I used Trapdoor for profiling. the profiling function profiles 100 ticks and gives you a breakdown of what part of the game uses more MSPT(miliseconds per tick). the limit is 50 MSPT, where the game drops below 20 TPS.

I created a new world in simulation 6, render 24. Placed a double chest with a hopper, and took these metrics with them empty:

[media]

5 mspt, which is pretty much just the player ticking the chunks around him.

after this, I filled the chest and hopper with shulker boxes filled with iron ingots (I discovered this issue in our iron farm)
This was the result:

[media]

36 MSPT only from ticking the chest and the hopper! I profiled multiple times, it kept giving me 40 mspt.

after breaking the hopper, I got this result:

[media]

as you can see, metrics went back to normal.

Shulker loaders are a very common storage solution in survival worlds, and a single full hopper pushing is enough to lag my very much overkill server! this could create enormous amounts of lag for slower servers!

ravinmaddhatter 2021-04-28T02:32:32Z

Most definitely still a problem i use this as my lag source when testing.

ravinmaddhatter 2021-05-31T23:40:20Z

I attached a behavior pack to make testing easier. This uses the test framework. But to test this use the following game test commands.

\gametest lag:check_lag

every time this runs it calculates the time between ticks.

\gametest lag:shulker_lag

This spawns a setup with shulker boxes that the player can turn the item in the item frame to see how the lag increases. On a single player world it takes a few of these setups depending upon the single threaded performance on a your processor. on a realm... best of luck, it will be unplayable.

Mike Ammerlaan 2021-06-21T18:23:46Z

MadHatter - sorry for the delay. I didn't see the GameTest attachment, can you try again?

ravinmaddhatter 2021-06-21T18:49:15Z

Attached.

Since we don't have access to the sleep time in bedrock (IE how much CPU time the game is taking up) i measure it using Date.now() in java script. If you run that command on consecutive ticks in the game test framework you SHOULD get 50ms. unless all of the Sleep time is used up. In the case of the shulker block lag issue you can use up all of the sleep time pretty easily causing major lag.

I know that on the other side of the Mojang "compile flags" settings wall the MSPT measurements would be actually trivial. But it is extremely hard to prove that something causes lag.... I have no way to measure how sleep time is impacted by things running in the game unless they are completely broken. But for now this one is so bad that even on an I9 with 32 gigs of ram i can lag my game with 6 hoppers and a few shulkers....

hhhxiao 2021-07-21T01:13:32Z

I recently did some tests when I was studying the problem of too high MSPT occupancy of the hopper. I used the hook function method to test the execution time of some hopper-related functions.

After observing the test results, I annotated a function called Hopper::_isFullContainer,In most static case(non item pull in or out), the main run time of hopper tick is on this function.

I know that your intention is to determine the hopper situation in advance, so as to reduce the execution of some codes. But unfortunately, your optimization did not have a very good effect due to the low execution efficiency of this function. When dealing with items with nbt, this mspt occupancy is more obvious.

So I hope you can try to optimize the code of this function to improve its execution efficiency.

Here are some of my test results:

conditions	hopper tick time(ms)	full check time(ms)
empy hopper	0.013	0.0
full hopper (5 full shulker box,same below)	0.884	0.872
full hopper points to full chest	5.289	5.287

silentwisperer 2021-08-07T19:17:14Z

5 full shulkers of enchanted items, pointing into a double chest with 1 item in each slot is also a great way to cause superb amounts of lag.

This bug really needs to be addressed, seeing as how most every storage system will have full hoppers pointing into full chests, at some point. Causing unnecessary lag and affecting gameplay. Because it doesn't affect fps, its nearly impossible for people to discover the source of the lag without previous knowledge of this bug.

GoldenHelmet 2021-08-17T12:40:26Z

@unknown: I have removed your comment and video because they are not relevant to this report. You have not understood what this report is about.

I have removed a lot of your comments from a lot of different reports over the last couple of months. Please make a better effort to fully understand reports. Do not comment unless you have new information to add. See our Bug Tracker Guidelines for guidance. Please use the Mojira Discord to discuss bugs, request update to reports, and ask questions.

I know English is not your first language. If you are not using a translation site/app you may want to consider using one.

Natthan 2021-09-14T22:36:06Z

Affects 1.17.11