Server Skin Problem

Here's the deal. Whenever you go through an end or nether portal on a server, your skin either changes to Steve or to a random person from the server.

-others still see your skin fine