0

We are currently facing an issue, where our hosts run out of memory. Only one single application is running on these hosts.

We first suspected the JVM to cause the problem and checked the actual mem usage. The JVM behaves fine, we are nearly 100% sure that there are no memory leaks (the mem graph showing free and used memory looks really fine, it's obvious when the GC kicks in, the avg mem usage us stable over a long period of time).

Still, the free mem on the host itself decreases slowly but steadily without indicating what process is chewing it up, each of the other services and daemons running on the host have a stable mem usage).

We still assume that something in our app causes this, but as the JVM looks well and our review of possible memory leaks didn't show any vulnerability we're kind of stuck now...

Any ideas?

Thank you very much for any hint what too look at!

ximarin
  • 401
  • 3
  • 14
  • First, check `ps aux --sort -rss` to verify if it is Java process that takes most physical memory. There are [many reasons](https://stackoverflow.com/a/36927242/3448419) why Java may consume much more than `-Xmx` value. Check [NativeMemoryTracking](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html) feature and native memory profilers like [jemalloc](http://jemalloc.net/) and [async-profiler](https://github.com/jvm-profiling-tools/async-profiler/) (`-e malloc` mode). – apangin Sep 23 '18 at 10:28
  • Stack Overflow is a site for programming and development questions. This question appears to be off-topic because it is not about programming or development. See [What topics can I ask about here](http://stackoverflow.com/help/on-topic) in the Help Center. Perhaps [Server Fault](http://serverfault.com/) or [Unix & Linux Stack Exchange](http://unix.stackexchange.com/) would be a better place to ask. – jww Sep 23 '18 at 16:37

1 Answers1

0

The most common reason for the memory to become used over time is the file cache holds more and more files are accessed. This isn't a problem as it's a read cache the memory can be dropped by the OS easily at any time.

e.g.

$ head -5 /proc/meminfo
MemTotal:       131751016 kB
MemFree:        109487136 kB
MemAvailable:   124110416 kB
Buffers:           78400 kB
Cached:         14764024 kB

This machine has plenty of free memory. But if I access lots of data.

$ find /opt -print | xargs wc -l
$ head -5 /proc/meminfo
MemTotal:       131751016 kB
MemFree:        105404804 kB
MemAvailable:   124107668 kB
Buffers:           78400 kB
Cached:         18824784 kB

Now there is 4 GB less free memory as the files are cached.

How long are they cached for? Until

  • There is a reason to reclaim the memory
  • the computer reboots
  • the drive is unmounted.

In short, the JVM could be accessing files and pulling them into a cache, but this doesn't mean it is a problem.

BTW An overnight backup of a server often fills the file cache, but this is normal.

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130