Only one player enters the entrence

When trying to enter the illusioners library/tower, only one player enters/dissapears and can not come back. The only solution to keep playing is to restart the game. The same occurs over and over again, if we try to enter.