I have a script that monitors the output of top and here is what the system looked like before java invoked the oom-killer:
TopLogger time: 2016-03-23:07:45:07.691254496
top - 07:45:07 up 1 day, 14:58, 2 users, load average: 1.60, 1.59, 1.61
Tasks: 179 total, 9 running, 170 sleeping, 0 stopped, 0 zombie
%Cpu(s): 8.4 us, 7.7 sy, 0.0 ni, 83.7 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem: 2038540 total, 1283840 used, 754700 free, 7372 buffers
KiB Swap: 0 total, 0 used, 0 free. 332372 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2336 root 0 -20 397256 162572 12568 S 131.0 8.0 2674:22 java
28845 odroid 20 0 4860 1056 728 R 31.4 0.1 0:00.10 top
1047 root 20 0 123136 22964 14236 S 10.5 1.1 113:30.21 Xorg
1181 root 20 0 12616 772 604 S 5.2 0.0 65:49.23 eGTouchD
21660 root 20 0 0 0 0 S 5.2 0.0 0:09.14 kworker/7:2
1 root 20 0 24052 3200 1808 S 0.0 0.2 0:28.11 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.44 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 3:40.20 ksoftirqd/0
As you can see java was consuming using 8% of the memory. The system itself had over 754 MB of memory available. At the same time, the kernel logs show that somehow this caused java to trigger the oom-killer:
Mar 23 07:45:12 odroid kernel: [140311.599648] [c1] java invoked oom-killer: gfp_mask=0x40d0, order=0, oom_score_adj=0
Mar 23 07:45:12 odroid kernel: [140311.599842] [c1] CPU: 1 PID: 3499 Comm: java Not tainted 3.10.96-78 #1
Mar 23 07:45:12 odroid kernel: [140311.599858] [c1] Backtrace:
Mar 23 07:45:12 odroid kernel: [140311.599897] [c1] [<c0012da4>] (dump_backtrace+0x0/0x114) from [<c0013014>] (show_stack+0x20/0x24)
Mar 23 07:45:12 odroid kernel: [140311.599912] [c1] r7:000040d0 r6:da80d600 r5:00000000 r4:da906000
Mar 23 07:45:12 odroid kernel: [140311.599960] [c1] [<c0012ff4>] (show_stack+0x0/0x24) from [<c062aff8>] (dump_stack+0x24/0x28)
Mar 23 07:45:12 odroid kernel: [140311.599983] [c1] [<c062afd4>] (dump_stack+0x0/0x28) from [<c062a520>] (dump_header+0x8c/0x244)
Mar 23 07:45:12 odroid kernel: [140311.600009] [c1] [<c062a494>] (dump_header+0x0/0x244) from [<c00ed684>] (oom_kill_process+0x364/0x3dc)
Mar 23 07:45:12 odroid kernel: [140311.600033] [c1] [<c00ed320>] (oom_kill_process+0x0/0x3dc) from [<c00edc20>] (out_of_memory+0x33c/0x390)
Mar 23 07:45:12 odroid kernel: [140311.600059] [c1] [<c00ed8e4>] (out_of_memory+0x0/0x390) from [<c00f241c>] (__alloc_pages_nodemask+0xb0c/0xb20)
Mar 23 07:45:12 odroid kernel: [140311.600083] [c1] [<c00f1910>] (__alloc_pages_nodemask+0x0/0xb20) from [<c00f2450>] (__get_free_pages+0x20/0x3c)
Mar 23 07:45:12 odroid kernel: [140311.600110] [c1] [<c00f2430>] (__get_free_pages+0x0/0x3c) from [<c04e7858>] (snd_malloc_pages+0x44/0x9c)
Mar 23 07:45:12 odroid kernel: [140311.600138] [c1] [<c04e7814>] (snd_malloc_pages+0x0/0x9c) from [<c04dcf90>] (snd_pcm_attach_substream+0x210/0x394)
Mar 23 07:45:12 odroid kernel: [140311.600154] [c1] r5:00000000 r4:dd90a600
Mar 23 07:45:12 odroid kernel: [140311.600193] [c1] [<c04dcd80>] (snd_pcm_attach_substream+0x0/0x394) from [<c04e2590>] (snd_pcm_open_substream+0x24/0xc4)
Mar 23 07:45:12 odroid kernel: [140311.600220] [c1] [<c04e256c>] (snd_pcm_open_substream+0x0/0xc4) from [<c04e26e8>] (snd_pcm_open+0xb8/0x208)
Mar 23 07:45:12 odroid kernel: [140311.600234] [c1] r5:dc68e000 r4:dd90a400
Mar 23 07:45:12 odroid kernel: [140311.600270] [c1] [<c04e2630>] (snd_pcm_open+0x0/0x208) from [<c04e28f0>] (snd_pcm_playback_open+0x4c/0x6c)
Mar 23 07:45:12 odroid kernel: [140311.600302] [c1] [<c04e28a4>] (snd_pcm_playback_open+0x0/0x6c) from [<c04d1dcc>] (snd_open+0xa8/0x1d4)
Mar 23 07:45:12 odroid kernel: [140311.600315] [c1] r5:dc68e000 r4:00000010
Mar 23 07:45:12 odroid kernel: [140311.600351] [c1] [<c04d1d24>] (snd_open+0x0/0x1d4) from [<c01396ac>] (chrdev_open+0x98/0x178)
Mar 23 07:45:12 odroid kernel: [140311.600367] [c1] r9:00000000 r8:da907e98 r7:de3a8080 r6:dc68e000 r5:dda90a20
Mar 23 07:45:12 odroid kernel: [140311.600367] r4:de3a8080
Mar 23 07:45:12 odroid kernel: [140311.600430] [c1] [<c0139614>] (chrdev_open+0x0/0x178) from [<c01337d0>] (do_dentry_open+0x1a4/0x26c)
Mar 23 07:45:12 odroid kernel: [140311.600443] [c1] r7:dc68e008 r6:c0139614 r5:dda90a20 r4:dc68e000
Mar 23 07:45:12 odroid kernel: [140311.600488] [c1] [<c013362c>] (do_dentry_open+0x0/0x26c) from [<c0133ba4>] (finish_open+0x44/0x5c)
Mar 23 07:45:12 odroid kernel: [140311.600501] [c1] r9:00000000 r8:da907e98 r7:00000802 r6:da907f60 r5:00000000
Mar 23 07:45:12 odroid kernel: [140311.600501] r4:da907e94
Mar 23 07:45:12 odroid kernel: [140311.600563] [c1] [<c0133b60>] (finish_open+0x0/0x5c) from [<c0142950>] (do_last+0x3d8/0xca0)
Mar 23 07:45:12 odroid kernel: [140311.600576] [c1] r5:00000000 r4:da907ee0
Mar 23 07:45:12 odroid kernel: [140311.600609] [c1] [<c0142578>] (do_last+0x0/0xca0) from [<c01432d8>] (path_openat+0xc0/0x4a0)
Mar 23 07:45:12 odroid kernel: [140311.600634] [c1] [<c0143218>] (path_openat+0x0/0x4a0) from [<c01444d0>] (do_filp_open+0x3c/0x98)
Mar 23 07:45:12 odroid kernel: [140311.600659] [c1] [<c0144494>] (do_filp_open+0x0/0x98) from [<c0134648>] (do_sys_open+0xf0/0x184)
Mar 23 07:45:12 odroid kernel: [140311.600672] [c1] r7:00000001 r6:dc46c000 r5:ffffff9c r4:00000047
Mar 23 07:45:12 odroid kernel: [140311.600718] [c1] [<c0134558>] (do_sys_open+0x0/0x184) from [<c0134704>] (SyS_open+0x28/0x2c)
Mar 23 07:45:12 odroid kernel: [140311.600730] [c1] r9:da906000 r8:c000e9a4 r7:00000005 r6:00000004 r5:ffffffff
Mar 23 07:45:12 odroid kernel: [140311.600730] r4:00000000
Mar 23 07:45:12 odroid kernel: [140311.600792] [c1] [<c01346dc>] (SyS_open+0x0/0x2c) from [<c000e780>] (ret_fast_syscall+0x0/0x38)
Mar 23 07:45:12 odroid kernel: [140311.600805] [c1] Mem-info:
Mar 23 07:45:12 odroid kernel: [140311.600820] [c1] Normal per-cpu:
Mar 23 07:45:12 odroid kernel: [140311.600835] [c1] CPU 0: hi: 186, btch: 31 usd: 0
Mar 23 07:45:12 odroid kernel: [140311.600850] [c1] CPU 1: hi: 186, btch: 31 usd: 31
Mar 23 07:45:12 odroid kernel: [140311.600866] [c1] CPU 2: hi: 186, btch: 31 usd: 0
Mar 23 07:45:12 odroid kernel: [140311.600880] [c1] CPU 3: hi: 186, btch: 31 usd: 0
Mar 23 07:45:12 odroid kernel: [140311.600895] [c1] CPU 4: hi: 186, btch: 31 usd: 59
Mar 23 07:45:12 odroid kernel: [140311.600910] [c1] CPU 5: hi: 186, btch: 31 usd: 30
Mar 23 07:45:12 odroid kernel: [140311.600925] [c1] CPU 6: hi: 186, btch: 31 usd: 0
Mar 23 07:45:12 odroid kernel: [140311.600940] [c1] CPU 7: hi: 186, btch: 31 usd: 0
Mar 23 07:45:12 odroid kernel: [140311.600952] [c1] HighMem per-cpu:
Mar 23 07:45:12 odroid kernel: [140311.600967] [c1] CPU 0: hi: 186, btch: 31 usd: 0
Mar 23 07:45:12 odroid kernel: [140311.600982] [c1] CPU 1: hi: 186, btch: 31 usd: 0
Mar 23 07:45:12 odroid kernel: [140311.600998] [c1] CPU 2: hi: 186, btch: 31 usd: 0
Mar 23 07:45:12 odroid kernel: [140311.601013] [c1] CPU 3: hi: 186, btch: 31 usd: 0
Mar 23 07:45:12 odroid kernel: [140311.601027] [c1] CPU 4: hi: 186, btch: 31 usd: 0
Mar 23 07:45:12 odroid kernel: [140311.601042] [c1] CPU 5: hi: 186, btch: 31 usd: 0
Mar 23 07:45:12 odroid kernel: [140311.601058] [c1] CPU 6: hi: 186, btch: 31 usd: 0
Mar 23 07:45:12 odroid kernel: [140311.601073] [c1] CPU 7: hi: 186, btch: 31 usd: 0
Mar 23 07:45:12 odroid kernel: [140311.601104] [c1] active_anon:83899 inactive_anon:4593 isolated_anon:0
Mar 23 07:45:12 odroid kernel: [140311.601104] active_file:41200 inactive_file:23749 isolated_file:0
Mar 23 07:45:12 odroid kernel: [140311.601104] unevictable:0 dirty:1757 writeback:4724 unstable:0
Mar 23 07:45:12 odroid kernel: [140311.601104] free:199161 slab_reclaimable:4434 slab_unreclaimable:104235
Mar 23 07:45:12 odroid kernel: [140311.601104] mapped:10688 shmem:5339 pagetables:703 bounce:4789
Mar 23 07:45:12 odroid kernel: [140311.601104] free_cma:91777
Mar 23 07:45:12 odroid kernel: [140311.601161] [c1] Normal free:259148kB min:2756kB low:3444kB high:4132kB active_anon:0kB inactive_anon:0kB active_file:128kB inactive_file:72kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:778240kB managed:474800kB mlocked:0kB dirty:16kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:17736kB slab_unreclaimable:416940kB kernel_stack:2856kB pagetables:2812kB unstable:0kB bounce:0kB free_cma:256516kB writeback_tmp:0kB pages_scanned:296 all_unreclaimable? yes
Mar 23 07:45:12 odroid kernel: [140311.601175] [c1] lowmem_reserve[]: 0 10128 10128
Mar 23 07:45:12 odroid kernel: [140311.601233] [c1] HighMem free:537496kB min:512kB low:2392kB high:4272kB active_anon:335596kB inactive_anon:18372kB active_file:164672kB inactive_file:94924kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1296384kB managed:1296384kB mlocked:0kB dirty:7012kB writeback:18896kB mapped:42752kB shmem:21356kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:876380kB free_cma:110592kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 23 07:45:12 odroid kernel: [140311.601247] [c1] lowmem_reserve[]: 0 0 0
Mar 23 07:45:12 odroid kernel: [140311.601280] [c1] Normal: 678*4kB (MC) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB (C) 0*1024kB 1*2048kB (C) 62*4096kB (C) = 259224kB
Mar 23 07:45:12 odroid kernel: [140311.601387] [c1] HighMem: 3433*4kB (UM) 1557*8kB (UM) 1098*16kB (UM) 930*32kB (UM) 308*64kB (UM) 82*128kB (UM) 21*256kB (UM) 7*512kB (UM) 3*1024kB (UM) 2*2048kB (M) 102*4096kB (MRC) = 537644kB
Mar 23 07:45:12 odroid kernel: [140311.601528] [c1] 70313 total pagecache pages
Mar 23 07:45:12 odroid kernel: [140311.601543] [c1] 0 pages in swap cache
Mar 23 07:45:12 odroid kernel: [140311.601558] [c1] Swap cache stats: add 0, delete 0, find 0/0
Mar 23 07:45:12 odroid kernel: [140311.601570] [c1] Free swap = 0kB
Mar 23 07:45:12 odroid kernel: [140311.601582] [c1] Total swap = 0kB
Mar 23 07:45:12 odroid kernel: [140311.616500] [c4] 518656 pages of RAM
Mar 23 07:45:12 odroid kernel: [140311.616515] [c4] 200048 free pages
Mar 23 07:45:12 odroid kernel: [140311.616522] [c4] 9021 reserved pages
Mar 23 07:45:12 odroid kernel: [140311.616530] [c4] 103836 slab pages
Mar 23 07:45:12 odroid kernel: [140311.616538] [c4] 1942637 pages shared
Mar 23 07:45:12 odroid kernel: [140311.616544] [c4] 0 pages swap cached
Mar 23 07:45:12 odroid kernel: [140311.616553] [c4] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Mar 23 07:45:12 odroid kernel: [140311.616587] [c4] [ 316] 0 316 1999 612 6 0 0 systemd-journal
Mar 23 07:45:12 odroid kernel: [140311.616599] [c4] [ 325] 0 325 3027 560 6 0 -1000 systemd-udevd
Mar 23 07:45:12 odroid kernel: [140311.616611] [c4] [ 518] 100 518 3067 270 5 0 0 systemd-timesyn
Mar 23 07:45:12 odroid kernel: [140311.616623] [c4] [ 885] 0 885 9410 802 12 0 0 accounts-daemon
Mar 23 07:45:12 odroid kernel: [140311.616635] [c4] [ 886] 105 886 1321 490 4 0 -900 dbus-daemon
Mar 23 07:45:12 odroid kernel: [140311.616645] [c4] [ 892] 0 892 8669 686 8 0 0 urfkilld
Mar 23 07:45:12 odroid kernel: [140311.616657] [c4] [ 896] 107 896 1084 381 5 0 0 avahi-daemon
Mar 23 07:45:12 odroid kernel: [140311.616669] [c4] [ 900] 107 900 1084 136 5 0 0 avahi-daemon
Mar 23 07:45:12 odroid kernel: [140311.616680] [c4] [ 903] 104 903 7787 464 8 0 0 rsyslogd
Mar 23 07:45:12 odroid kernel: [140311.616692] [c4] [ 909] 0 909 10603 1716 14 0 0 ModemManager
Mar 23 07:45:12 odroid kernel: [140311.616703] [c4] [ 914] 108 914 11270 1335 16 0 0 whoopsie
Mar 23 07:45:12 odroid kernel: [140311.616714] [c4] [ 918] 0 918 2213 1324 7 0 0 ofonod
Mar 23 07:45:12 odroid kernel: [140311.616725] [c4] [ 923] 0 923 866 355 3 0 0 systemd-logind
Mar 23 07:45:12 odroid kernel: [140311.616737] [c4] [ 929] 0 929 18289 1809 18 0 0 NetworkManager
Mar 23 07:45:12 odroid kernel: [140311.616749] [c4] [ 941] 0 941 876 246 3 0 0 cgmanager
Mar 23 07:45:12 odroid kernel: [140311.616760] [c4] [ 950] 0 950 1324 453 4 0 0 bluetoothd
Mar 23 07:45:12 odroid kernel: [140311.616771] [c4] [ 958] 0 958 1222 242 5 0 0 cron
Mar 23 07:45:12 odroid kernel: [140311.616782] [c4] [ 995] 0 995 2064 595 6 0 -1000 sshd
Mar 23 07:45:12 odroid kernel: [140311.616793] [c4] [ 999] 0 999 8325 1068 13 0 0 cups-browsed
Mar 23 07:45:12 odroid kernel: [140311.616805] [c4] [ 1000] 0 1000 8696 715 11 0 0 polkitd
Mar 23 07:45:12 odroid kernel: [140311.616816] [c4] [ 1014] 0 1014 10947 745 12 0 0 lightdm
Mar 23 07:45:12 odroid kernel: [140311.616827] [c4] [ 1033] 0 1033 1710 461 5 0 0 wpa_supplicant
Mar 23 07:45:12 odroid kernel: [140311.616838] [c4] [ 1047] 0 1047 30784 5741 36 0 0 Xorg
Mar 23 07:45:12 odroid kernel: [140311.616849] [c4] [ 1178] 112 1178 1635 245 6 0 0 kerneloops
Mar 23 07:45:12 odroid kernel: [140311.616860] [c4] [ 1181] 0 1181 3154 193 5 0 0 eGTouchD
Mar 23 07:45:12 odroid kernel: [140311.616872] [c4] [ 1225] 0 1225 1006 179 5 0 0 agetty
Mar 23 07:45:12 odroid kernel: [140311.616883] [c4] [ 1229] 0 1229 962 175 4 0 0 agetty
Mar 23 07:45:12 odroid kernel: [140311.616893] [c4] [ 1384] 0 1384 894 302 4 0 0 master
Mar 23 07:45:12 odroid kernel: [140311.616904] [c4] [ 1386] 114 1386 922 296 4 0 0 qmgr
Mar 23 07:45:12 odroid kernel: [140311.616915] [c4] [ 1449] 0 1449 6651 720 8 0 0 lightdm
Mar 23 07:45:12 odroid kernel: [140311.616926] [c4] [ 1475] 1000 1475 1602 822 5 0 0 systemd
Mar 23 07:45:12 odroid kernel: [140311.616936] [c4] [ 1480] 1000 1480 1748 387 6 0 0 (sd-pam)
Mar 23 07:45:12 odroid kernel: [140311.616947] [c4] [ 1485] 1000 1485 3448 1337 9 0 0 fluxbox
Mar 23 07:45:12 odroid kernel: [140311.616958] [c4] [ 1534] 1000 1534 726 243 4 0 0 unclutter
Mar 23 07:45:12 odroid kernel: [140311.616969] [c4] [ 1539] 1000 1539 731 83 3 0 0 ssh-agent
Mar 23 07:45:12 odroid kernel: [140311.616979] [c4] [ 1542] 1000 1542 1226 185 5 0 0 dbus-launch
Mar 23 07:45:12 odroid kernel: [140311.616991] [c4] [ 1543] 1000 1543 1213 199 4 0 0 dbus-daemon
Mar 23 07:45:12 odroid kernel: [140311.617001] [c4] [ 1559] 1000 1559 1090 309 4 0 0 d4voip.sh
Mar 23 07:45:12 odroid kernel: [140311.617013] [c4] [ 1561] 1000 1561 3136 1559 8 0 0 idesk
Mar 23 07:45:12 odroid kernel: [140311.617023] [c4] [ 1637] 1000 1637 1163 379 5 0 0 TopClientLogger.bsh
Mar 23 07:45:12 odroid kernel: [140311.617034] [c4] [ 1638] 1000 1638 1267 243 5 0 0 logger
Mar 23 07:45:12 odroid kernel: [140311.617044] [c4] [ 1703] 0 1703 3034 801 8 0 0 sshd
Mar 23 07:45:12 odroid kernel: [140311.617055] [c4] [ 1883] 1000 1883 3068 409 8 0 0 sshd
Mar 23 07:45:12 odroid kernel: [140311.617066] [c4] [ 1894] 1000 1894 1206 403 5 0 0 bash
Mar 23 07:45:12 odroid kernel: [140311.617077] [c4] [ 2014] 0 2014 1683 400 5 0 0 sudo
Mar 23 07:45:12 odroid kernel: [140311.617088] [c4] [ 2021] 0 2021 1586 353 6 0 0 su
Mar 23 07:45:12 odroid kernel: [140311.617099] [c4] [ 2030] 0 2030 1243 444 5 0 0 bash
Mar 23 07:45:12 odroid kernel: [140311.617111] [c4] [ 2290] 1000 2290 93900 14651 55 0 0 java
Mar 23 07:45:12 odroid kernel: [140311.617122] [c4] [ 2291] 1000 2291 93009 13010 51 0 0 java
Mar 23 07:45:12 odroid kernel: [140311.617133] [c4] [ 2292] 1000 2292 92993 13138 52 0 0 java
Mar 23 07:45:12 odroid kernel: [140311.617145] [c4] [ 2293] 0 2293 1683 400 5 0 0 sudo
Mar 23 07:45:12 odroid kernel: [140311.617158] [c4] [ 2336] 0 2336 99314 40643 110 0 0 java
Mar 23 07:45:12 odroid kernel: [140311.617170] [c4] [28839] 0 28839 2867 733 7 0 0 cupsd
Mar 23 07:45:12 odroid kernel: [140311.617182] [c4] [28841] 7 28841 2131 523 7 0 0 dbus
Mar 23 07:45:12 odroid kernel: [140311.617193] [c4] [14795] 0 14795 3034 760 8 0 0 sshd
Mar 23 07:45:12 odroid kernel: [140311.617204] [c4] [14910] 1000 14910 3067 412 7 0 0 sshd
Mar 23 07:45:12 odroid kernel: [140311.617215] [c4] [14918] 1000 14918 454 182 3 0 0 sftp-server
Mar 23 07:45:12 odroid kernel: [140311.617230] [c4] [24925] 114 24925 913 287 4 0 0 pickup
Mar 23 07:45:12 odroid kernel: [140311.617243] [c4] [27246] 0 27246 446 190 3 0 0 anacron
Mar 23 07:45:12 odroid kernel: [140311.617255] [c4] [27767] 0 27767 358 114 2 0 0 sh
Mar 23 07:45:12 odroid kernel: [140311.617266] [c4] [27768] 0 27768 341 112 3 0 0 run-parts
Mar 23 07:45:12 odroid kernel: [140311.617276] [c4] [27775] 0 27775 358 126 3 0 0 apt
Mar 23 07:45:12 odroid kernel: [140311.617290] [c4] [28849] 1000 28849 855 122 4 0 0 sleep
Mar 23 07:45:12 odroid kernel: [140311.617300] [c4] [28883] 0 28883 1865 673 6 0 0 apt-get
Mar 23 07:45:12 odroid kernel: [140311.617311] [c4] [28886] 0 28886 1749 497 6 0 0 http
Mar 23 07:45:12 odroid kernel: [140311.617322] [c4] [28887] 0 28887 1750 523 6 0 0 http
Mar 23 07:45:12 odroid kernel: [140311.617334] [c4] [28890] 0 28890 1628 397 5 0 0 gpgv
Mar 23 07:45:12 odroid kernel: [140311.617344] [c4] [28895] 0 28895 1626 447 5 0 0 copy
Mar 23 07:45:12 odroid kernel: [140311.617354] [c4] Out of memory: Kill process 2336 (java) score 77 or sacrifice child
Mar 23 07:45:12 odroid kernel: [140311.623829] [c4] Killed process 2336 (java) total-vm:397256kB, anon-rss:150004kB, file-rss:12568kB
Is our application leaking memory? Or do we need more then 2GB of memory on the system?
Thanks, any help is appreciated.
