9

This answer describes the out of memory killer on Linux.

Is there a way to get notified when the OOM killer has killed something? Email is preferable. One might do this with a watchdog, but I wonder what the easiest and most robust way is.

Community
  • 1
  • 1
dfrankow
  • 20,191
  • 41
  • 152
  • 214

2 Answers2

3

The "logwatch" program will periodically mail you all log lines it's configured to be interested in.

It can be configured to email you about an OOM (unless it runs out of memory itself).

Email notifications aren't a good substitute for proper monitoring from an external system as a machine which has no free memory will not normally be able to send email.

Also, if your system is important, then email is probably too slow to get a response from Ops (especially out of hours)

MarkR
  • 62,604
  • 14
  • 116
  • 151
  • *a machine which has no free memory will not normally be able to send email* – the process could be killed because it exceeded its own limit of memory (think *cgroups*) not necessarily because the system as a whole is out of memory. – Piotr Dobrogost Nov 06 '18 at 10:58
  • One can also use `/proc//oom_score_adjust` to volunteer a process for killing to keep the system running. If you want to be able to cope with system running out of memory, you have to figure out which processes you prefer to kill in case something must go. That way your watchdog process can still send email after the volunteer process has been killed and maybe try automatically restarting the killed process later (assuming that the out of RAM situation was only temporary). – Mikko Rantalainen Aug 22 '21 at 15:59
0

You can read and interpret the kernel log, as syslog records it to /var/log/, and act upon it.

user562374
  • 3,817
  • 1
  • 22
  • 19