Hoppers pushing full shulker boxes into a shulker box or items into full chests causes server lag

I was unable to reproduce this in a small test world (on an iPhone 8 running iOS 13.4.1). I filled 5 shulker boxes with cobblestone and put them into a hopper pointed into another shulker box. Animals and falling sand appeared to move at normal speeds.

Is it possible that the lag you experienced is, instead, the accumulation of multiple resource-demanding events taking place in your world, and the shulker-hopper setup you made was just the last straw? Would you be able to provide a video or test world?

did you try on a realm or server? 20 shulkers in this config will cause item transfer rate to reduce to 1 item every 4 seconds? I did put server in the title.

I have uploaded the world, upload the attached world file to a realm or server. i will upload a repro video to youtube in a few minutes and attached the link.

https://youtu.be/eTPg_hXs8hM

This video explains how easy this is to reproduce. I didnt put it in the BDS or Realms pool because it impacts both. a Jira Admin can move it if they choose.

Thank you for the additional information. I didn't realize you mean BDS/Realm because the local game also has client and server sides and their interaction plays a role in some bugs.

I was also able to reproduce in a single player world, it just takes far more hoppers. with 30 hoppers, my computer is heavilly bogged down. Realms are just really really weak and susceptible to lag.

The faster your computer is the more shulkers pushed into hoppers are required.

Thanks again for the update. Confirmed in 1.14.60. On my iPhone 8 I started to notice slight lag with 8 hoppers trying to push full Shulkers, and by 14 it was extreme.

One note: 

After playing around with this a fair bit, i believe the important data in this ticket is as follows:

• Full shulker's have more impact than empty shulker

• lag is reduced when the hopper target is full

• CPU read write speed seems to impact repro (IE realms and low end servers are more susceptible)

From that information, i believe this implies the server is copying the data for the shulker transfer, including all of its stored data, before checking the container type. However it does appear to check the container "fullness" before attempting the push. If the container type was checked at the same time the container fullness the data transfer would not occur and the lag would be reduced. 

This may an over simplification of the algorithm, however it from testing that seems to be what is happening.

If it stays in, i know this can be exploited as a lag switch, it can be used to do things like obtain objects intended to be unattainable, duplication, and other game breaking exploits. 

Note: seems to be reproducible with the equivalent number of hoppers pointing into full chests as items in shulker boxes. This may seem confusing but here is the scenario

54 hoppers facing into 54 full chests all full of items, you can get equivalent lag to 4 shulkers on a server/realm. The lag caused is way more several people being online. it does bring our server to it knees. 

this bug is considerably worse than we thought, or at least it has gotten really bad in .210 for some reason.
my server specs: Ryzen 2700@4ghz, 16 gigs of ram@3000cl18, running Trapdoor for profiling, Windows BDS for .210, locally, max threads 0 (or as many as it can use).
for this test I used Trapdoor for profiling. the profiling function profiles 100 ticks and gives you a breakdown of what part of the game uses more MSPT(miliseconds per tick). the limit is 50 MSPT, where the game drops below 20 TPS.

I created a new world in simulation 6, render 24. Placed a double chest with a hopper, and took these metrics with them empty:

5 mspt, which is pretty much just the player ticking the chunks around him.

after this, I filled the chest and hopper with shulker boxes filled with iron ingots (I discovered this issue in our iron farm)
This was the result:

36 MSPT only from ticking the chest and the hopper! I profiled multiple times, it kept giving me 40 mspt.

after breaking the hopper, I got this result:

as you can see, metrics went back to normal.

Shulker loaders are a very common storage solution in survival worlds, and a single full hopper pushing is enough to lag my very much overkill server! this could create enormous amounts of lag for slower servers!

 Most definitely still a problem i use this as my lag source when testing.

I attached a behavior pack to make testing easier. This uses the test framework. But to test this use the following game test commands.

\gametest lag:check_lag

every time this runs it calculates the time between ticks.

\gametest lag:shulker_lag

This spawns a setup with shulker boxes that the player can turn the item in the item frame to see how the lag increases. On a single player world it takes a few of these setups depending upon the single threaded performance on a your processor. on a realm... best of luck, it will be unplayable. 

MadHatter - sorry for the delay. I didn't see the GameTest attachment, can you try again?

Attached.

Since we don't have access to the sleep time in bedrock (IE how much CPU time the game is taking up) i measure it using Date.now() in java script. If you run that command on consecutive ticks in the game test framework you SHOULD get 50ms. unless all of the Sleep time is used up. In the case of the shulker block lag issue you can use up all of the sleep time pretty easily causing major lag.

I know that on the other side of the Mojang "compile flags" settings wall the MSPT measurements would be actually trivial. But it is extremely hard to prove that something causes lag.... I have no way to measure how sleep time is impacted by things running in the game unless they are completely broken. But for now this one is so bad that even on an I9 with 32 gigs of ram i can lag my game with 6 hoppers and a few shulkers....

I recently did some tests when I was studying the problem of too high MSPT occupancy of the hopper. I used the hook function method to test the execution time of  some hopper-related functions.

After observing the test results, I annotated a function called Hopper::_isFullContainer,In most static case(non item pull in or out), the main run time of hopper tick is on this function.

I know that your intention is to determine the hopper situation in advance, so as to reduce the execution of some codes. But unfortunately, your optimization did not have a very good effect due to the low execution efficiency of this function. When dealing with items with nbt, this mspt occupancy is more obvious.

So I hope you can try to optimize the code of this function to improve its execution efficiency. 

Here are some of my test results:

5 full shulkers of enchanted items, pointing into a double chest with 1 item in each slot is also a great way to cause superb amounts of lag.

This bug really needs to be addressed, seeing as how most every storage system will have full hoppers pointing into full chests, at some point. Causing unnecessary lag and affecting gameplay. Because it doesn't affect fps, its nearly impossible for people to discover the source of the lag without previous knowledge of this bug.

@unknown: I have removed your comment and video because they are not relevant to this report. You have not understood what this report is about.

I have removed a lot of your comments from a lot of different reports over the last couple of months. Please make a better effort to fully understand reports. Do not comment unless you have new information to add. See our Bug Tracker Guidelines for guidance. Please use the Mojira Discord to discuss bugs, request update to reports, and ask questions.

I know English is not your first language. If you are not using a translation site/app you may want to consider using one.

Affects 1.17.11