Can't join friends worlds with add ons

I'm unable to join my friends when they have add ons active 90% of the time. When they disable the shared packs I always get the “failed to connect to world) when shared packs is on, the download stops halfway and then disconnects me. Some people have this issue when joining me especially when I use specific packs. However my wifi is 1gb speeds and pretty crystal service.