Finding which process was killed by Linux OOM killer

Question

When Linux runs out of memory (OOM), the OOM killer chooses a process to kill based on some heuristics (it's an interesting read: http://lwn.net/Articles/317814/).

How can one programmatically determine which processes have recently been killed by the OOM killer?

score 237 · Answer 1 · edited Sep 02 '21 at 18:40

237

Try this so you don't need to worry about where your logs are:

dmesg -T | egrep -i 'killed process'

-T, --ctime - Print human-readable timestamps.

edited Sep 02 '21 at 18:40

Elijah Lynn

12,272
10
61
91

answered Apr 11 '13 at 15:58

Jose Fernandez

2,501
1
14
4

1

This is also useful, but while I unfortunately can't explain it, I'm seeing results in `/var/log/messages` that aren't showing up in `dmesg`/`/var/log/dmesg`. It could be some sort of misconfiguration, but worth noting that using both approaches could be a good idea. – kungphu Mar 01 '16 at 01:21
3

Not sure about your log file, but the output of dmesg is from a limited-size ring buffer. If other things have filled the buffer since the oom-killer then you'll lose the oom-killer output. – Dan Pritts Apr 11 '16 at 16:21
This was the only way I found how to see that process was killed in OpenVZ container – igo Oct 16 '16 at 08:23
3

Compared to /var/log/messages, this has the advantages of not requiring root privileges – Kineolyan Jan 29 '18 at 13:18
For me it was just at the bottom of dmesg -T. I didn't need to grep. – fei0x Dec 13 '21 at 19:14

score 209 · Accepted Answer · edited Nov 10 '16 at 00:49

209

Try this out:

grep -i 'killed process' /var/log/messages

edited Nov 10 '16 at 00:49

antak

19,481
9
72
80

answered Mar 09 '09 at 02:54

John Feminella

303,634
46
339
357

22

FWIW, I get those messages in syslog, or kern.log, but not /var/log/messages – jberryman Nov 22 '11 at 20:07
49

You can use "egrep -i -r 'killed process' /var/log/" to search it also in other places. – metdos Dec 27 '11 at 07:22
8

@jberryman: For some reason, syslog is in `/var/log/syslog` on some distros, and `/var/log/messages` on others. I think it's Debian for the former and Red Hat for the latter, BICBW. – Tom Anderson Mar 26 '13 at 11:34
7

" dmesg | egrep -i 'killed process' " and you can search logs anywhere (including archived ones) :) – Phil B Aug 16 '14 at 11:04
3

`egrep` doesn't make sense here. Plain old `grep`, or if we're being specific, `fgrep`, makes much more sense. (Editing answer accordingly.) – antak Nov 10 '16 at 00:49
1

The precise location depends on the logging arcitecture and its configuration. `/etc/syslog.conf` is a good place to start looking, though on some arcane systems, this too could be in a different location. – tripleee Feb 11 '17 at 15:23

score 59 · Answer 3 · edited Feb 24 '20 at 12:36

59

Now dstat provides the feature to find out in your running system which process is candidate for getting killed by oom mechanism

dstat --top-oom
 --out-of-memory---
  kill score
 java           77
 java           77
 java           77

and as per man page

  --top-oom
          show process that will be killed by OOM the first

edited Feb 24 '20 at 12:36

laurent

88,262
77
290
428

answered Sep 02 '14 at 11:32

Prashant Lakhera

850
7
13

2

This info is meaningless without knowing what the score means, and that's not documented anywhere. All you might see is the score increase, then the process being killed, so maybe it was the oom killer, or maybe it was something else, there's no way to be sure. – laurent Feb 24 '20 at 12:41
Where does `dstat` read this kill score data from? (A file?) Edit: Just found: `/proc//oom_score` – Dentrax Nov 10 '22 at 11:52

score 24 · Answer 4 · edited Mar 26 '13 at 05:37

24

Try this out:

grep "Killed process" /var/log/syslog

edited Mar 26 '13 at 05:37

Javier

12,100
5
46
57

answered Mar 26 '13 at 05:18

Praveen

241
2
5

Finding which process was killed by Linux OOM killer

4 Answers4

Linked