I am seeing a similar issue on my server. I'm running version 1.7.2 and I'm seeing overly large memory usage.
We have been playing without trouble for a week, and then someone started exploring the nether. Immediately the memory usage shot up to all of the ram on the server (3 gigs) for only 3 people logged in and standing in the same house.
We tried restarting the server and logging back in, I can watch top as we log in. The memory usage skyrockets to the maximum available as soon as the three of us log in.
Two of us were able to play for a few minutes with the ram usage at a 2-3 hundred megabytes, then when the third person logged in it was instantly 3 gigs.
The logs don't show anything unusual other than the "can't keep up" messages that result from running out of ram.
This is the command as run through init.d.
java -Xmx2048M -Xms2048M -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalPacing -XX:ParallelGCThreads=1 -XX:+AggressiveOpts -jar minecraft_server.jar nogui
Updated to OpenJDK 7 and the problem seems to have been fixed.
Thanks!