Cannot play with friend

A couple of months back my friend and I stopped being able to play together. It would tell us both "Unable to Connect to World." We could still play together on other worlds and servers but not eachother's worlds. We have both gone through the settings but couldn't find anything that might cause this.