I'm also experiencing the same problem. I have a port of Technoblade's texture pack, and 9 other overlay texture packs including FoxyNoTail ones. I was playing today, when I realized there was a new update! So I went to the Microsoft Store to download it, and when I returned, that distracting little arrow was there! I thought it was one of my texture packs auto updating, but I checked, and it happened for all of them.
I am so sorry, but I figured out my problem. This isn't a Minecraft bug, it's a behavior pack config issue. I don't know how to take this post down though...😞