When ever I load the server list, it takes a good 10-15 seconds for any info to load. Afterwards, if I join a server it also locks up for 10-15 seconds when joining a server.
In 1.3.1/2, there was this (http://www.minecraftforum.net/topic/1461783-fixing-lag-and-freezing-when-joining-a-mp-server/) patch that fixed it. While I don't know the technical reasons very well, I do know that this fixed my issue. Hopefully that can help you guys track it down.
This ticket has been marked as a duplicate of MC-2257 , in favor of the better description/information provided.