No Memory, Super low FPS

When I log onto the multiplayer server the fps is in the low 20s but gradually decreases until it reaches 4 or 5 fps. This usually takes about a minute or two at the most. If I quit the game and log back in I have another minute to play. But after logging out and back in a few times the game will boot me off saying that there is not enough memory for the game. The game is playing at low memory levels for some unknown reason as well. The used memory is only slightly lower than the allocated memory, which is at most about 15%.