Excessive Server Memory usage ?

I will have to monitor my server more closely before I can say anything for sure.

Ive had to disable watchdog, as it was crashing the server even with a restart set at 1hr intervals, before that it would lag tot he point where about half the users were kicked off the server while others werent before itd eventually crash, presently ive got restart times set at 40min and now 1.8 is released my avg user count is around 20 to 60 an nearly always using 10+ GB memory

From what I've seen the memory does seem to increase but it's to early for me to say if it's leaking, because the memory increase is very little/slow so far.

The reserved size for the process does indeed slowly increase:
Day 1 of a newly started server: VmRSS: 1197968 kB
End of day 2 of a newly statrted server: VmRSS: 1213656 kB

This is only with two people playing.for a few hours.
Java memory is set to -Xmx1024M -Xms1024M -oss4M -ss4M

Here's another measurement from the two days:
Day one: 1.14 gb + 198.00 kb = 1.14 gb java
Day two: 1.16 gb + 131.00 kb = 1.16 gb java
(first colum is private memory, second is shared, third is total)

Maybe not too alarming yet but if it continue doing this, then I think we have a problem,

I just checked my server. After running for about 10hs, with a maximum of 3 persons online and 1 currently, it stands at 4500mb out of 5000mb used.
Something nasty is definetly going on. It just cant be possible that a server keep draining memory without releasing at least some.

Also, i checked the entities IDs, with fresh start, the entities IDs (seen on the log when someone joins the server) are usually pretty low. Between 500-1000. But if you check after some hours, the entities IDs rise to 100.000 and even more.

Are servers not unloading and disposing entities correctly?

I think entity IDs always increases, I checked some older logs and it's the same behaviour.

End of day 3:
VmRSS: 1218240 kB

Other memory mesurement:
1.16 gb + 111.00 kb = 1.16 gb

As you can see, memory usage is actually less than day 2 now. No crashes so far either.
I also have a third "live" memory usage test running and that seems to clean up fine too.

EDIT:
For your server with about 20 players you should be fine with allocating about 1-2Gb of ram to java (2Gb if you have 40 players online), have you tried that? Instead of giving it 5Gb.

I've tried that, memory filled up in less than half an hour.
I will post my server status with 3 players online right now in a sec.

I've restarted the server 45 minutes ago aprox.
This is the status right now... more than half of the memory used already, and none being returned to the system.

Well, I also have a memory graph in for minecraft on a webpage, that is always at 100% (even before i started using 1.8) so in my case that's normal. Only time when its less is when I just started the server. Bear in mind that the server might do internal caching that's why it fills up. Does it ever go beyond your 5Gb allocation?

I am working with the person who reported this (Warren Liddell) on his server. On a restart, we watched it go from 25% usage to, 30 minutes later, 77% usage. (Measured using "top" in Ubuntu). We're having to restart MC every 30m to keep it from consuming all system memory.
That server is running Ubuntu, kernel 3.10.23-xxxx-std-ipv6-64, java version "1.7.0_55"

I set up a test system with Debian 7 3.2.0-4-amd64 kernel, java version "1.6.0_26" 3gb ram and 2gb allocated to Minecraft 1.8. Vanilla, just the plain download available from Mojang. Found that memory usage jumped 1% every time myself and another player logged out and back in. It seems that memory is being allocated for the user session but is not be deallocated when the users leave. Additionally, memory usage increased even at idle with one user logged in, and not doing anything. Because Warren's server is extremely popular (currently with a very high google ranking) and has hundreds of users logging in and out daily, the problem is only made worse.

To do more experimenting and test out this bug, we installed 1.8 on a CentOS 6.5 server with 2.6.32-431.11.2.el6.x86_64 #1 SMP kernel and Java version "1.7.0_55" which matches that being used by Warren's server. With 5 different players logging in/out in rapid succession, playing for about 10 minutes, and then logging in/out again in rapid succession (about 5 times each, both times) the ram usage started at 10.9% and never went above 15.3%.

This would indicate to me that the issue is replicated when using a version 3.x Linux kernel on a Debian related OS, however I do not have a CentOS with 3.x kernel to test with. I will report back when I have had the opportunity to test it in a VM.

@Ryan I have a MineOS server which I belive is Debian based with kernel 3.x
uname -r
3.2.0-4-amd64

I can confirm that this problem is hapenning to me, not sure if thats what u wanted to check.

For the record I have Gentoo running with Linux 3.16.0 #1 SMP PREEMPT, I THOUGHT the minecraft server was behaving itsellf because I've seen it release memory but now when I checked it's using 10Mb more than yesterday.

I don't have that many users logging in and out, so I can't say anything about it but considering the java process memory grows slowly for me Ryan might be onto something here.
When I initially started to monitor the minecraft/java process it used 1.14Gb of memory as seen in my previous comment, it's now using 1.17Gb.

We've changed the system over to CentOS 6.5 Kernel 2.6.32-431.23.3.el6.x86_64 to do some further testing.

It made very minimal change it was barely noticeable after 24hrs

So it consumes less memory with 2.6.32 or what are you saying?

I don't get the point here.
The JVM consumes of course all assigned memory.

Example @unknown:

Java memory is set to -Xmx1024M -Xms1024M -oss4M -ss4M
Day one: 1.14 gb + 198.00 kb = 1.14 gb java
Day two: 1.16 gb + 131.00 kb = 1.16 gb java

You assign 1 GB to JVM heap and it uses 1 GB + stack + JVM itself

That's to be expected.
If you assign 16 GB heap, all the 16 GB heap is allocated before the GC kicks in, that's no mem leak, that's just the way the JVM works.

@unknown

I run a 1.8 server which started as a 4GB server with avg of 20 users an found it wasnt enough

What's the meaning of "found it wasnt enough" ? did you get exceptions java.lang.OutOfMemoryError: Java heap space ? If not, then it was enough.

The server was constantly using the entire amount of memory allocated to the server untill i stopped at 16GB which the server uses all the available memory usually between 1-2 hrs of being operational and the Block Lag & Server Tick takes to long an console starts screaming and eventually crashes due to watchdog or someone restarts the server

Yeah, I know what you mean Kumasasa but some seem to have memory issues beyond the set heap.
I haven't really gotten a straight answer from anyone else what happens if they set their heap size to something more sane. For 120 players about 6Gb should be enough, like Kumasasa said, if it still goes beyond that and crashes with OOM message, there is a problem.

Also don't forget to provide a crash log.

Maybe you should tune the JVM arguments a bit, at least the JVM argument -server should be used.
Quick search: finds e.g. this http://www.minecraftforum.net/forums/support/server-support/1886149-modded-server-jvm-arguments

I will give this a try and see what happens.

Kumasasa, running the server with the default Arguments (published with the server JAR file in the minecraft.net site) gives the same results.

I've spent hours trying different parameters, an ALL of them give the same result: heap usage increases, but NEVER decreases.

@@unknown: As said before: An increase of memory up to the limits specified in the JVM arguments normal.
This ticket is about @unknown's problems

the Block Lag & Server Tick takes to long an console starts screaming and eventually crashes due to watchdog

the description of the problem seems to fail to mention what you are saying.

For what I can understand from it, its the same problem I'm having with my server since 1.8 came out.

And again, we are not talking about an "increase of memory" we are talking of a hungy beast memory server which uses and colapses the system after consuming all its resources. Im not a computer expert, but I've studied and researched memory usage, the system takes after no longer needed its released. Its a cycle... which is not being done with the minecraft servers since 1.8 (prereleases).

If there is new information about the ticket that might point this problem in another direction it would be nice to have it in the description.

Alejandro is correct. This is an issue with memory being allocated and not released. I have tested this on three different kernels (3.2, 2.6.32, 3.10.23) and the issue exists on all of them. Last night the server started increasing memory usage 1-2 percent per second until I restarted it, and then it stabilized. This is with the exact same number of users on the server as before the issue. Alejandro is also correct in that this problem started specifically on release 1.8 Pre1. Prior to that it was relatively stable. So yes, this is a memory issue. As a professional Linux server admin, I stepped in to help Warren with this problem. We've determined that the block lag is happening when the Java process is using all of its allocated memory. Unfortunately once it reaches its maximum (around 97.5%) it stays there with no let up, and everyone on the server (usually 15-30 people, as we have lost players due to this) starts lagging, rubber banding, block lagging, and complaining. A restart solves it for a few minutes until the memory usage starts climbing as players log in.

This issue is repeatable. We even changed OS's on the server (Ubuntu to CentOS). The same server on earlier releases had no issues on 4GB memory. There is no good reason for this to be requiring so much memory. I can only imagine that there is a memory leak at work here. I'm sure Warren would welcome one of Mojang's staff to examine the server and see the issue first hand.

My server is now using 1.19Gb RSS memory... a few days ago it was 1.14.
I have given java 1Gb in heap... so why is it still slowly increasing?

To those who wonder what RSS is in linux:
RSS is the Resident Set Size and is used to show how much memory is allocated to that process and is in RAM. It does not include memory that is swapped out. It does include memory from shared libraries as long as the pages from those libraries are actually in memory. It does include all stack and heap memory.

Now it's using 1.22Gb and my servers (the machine itself) memory statistics graph is looking weird.

I think its time for the mojang team to take over this issue. This is a very serious problem, if the app is leaking memory, its stability is at risk.

I've already said that if needed I can provide access to a remote linux console to the server, and any access needed to investigate the issue.

For the mod: please take this as a real problem, there is already enough evidence in this ticket to move to confirm and its still not happening. Why is this it?

Done.

Ive changed my JVM settings as advised to include usage of the -server tag option and i have seen no difference in consumption of memory etc

following (experiencing the same memory leak issues & my ticket was closed as a dupe)

Well, it's a memory leak, for me it's very slow but it's there., I'm pretty sure now. It's up to Mojamg to find it, I can't say what's causing it. 😞

EDIT:
The minecraft server process is now using swapped memory too. Memory usage is still increasing but,it's slightly harder to follow now because of the swapped out memory.