1

I want to be able to monitor kernel panics - know if and when they have happened.
Is there a way to know, after the machine has booted, that it went down due to a kernel panic (and not, for example, an ordered reboot or a power failure)?

The machine may be configured with KDUMP and/or KDB, but I prefer not to assume that either is or is not installed.

Patching the kernel is an option, though I prefer to avoid it. But even if I do it, I'm not sure what can the patch do.

I'm using kernel 2.6.18 (ancient, I know). Solutions for newer kernels may be interesting too.

Thanks.

ugoren
  • 16,023
  • 3
  • 35
  • 65
  • Do you have any facility(H/W UART logging etc..) to take logs using console utilities - teraterm, putty ..?. – kzs Jan 02 '14 at 09:31
  • @kzs, In general, no. I want to collect the information on the same machine, and examine it somehow later. – ugoren Jan 02 '14 at 12:04
  • possible duplicate of http://stackoverflow.com/questions/13189576/getting-linux-kernel-debug-information-after-kernel-crash – Jeyaram Jan 02 '14 at 12:22
  • @Jeyaram, very similar indeed, somehow I missed that one. However, the best advice there is `/var/log/messages`, and I'm really not sure you always get the crash messages there. – ugoren Jan 02 '14 at 12:36
  • try this, sudo cat /proc/kmsg | grep -w panic , should be able to see panic messages. panic messages are reported using emergency flag (KERN_CRITICAL), so it should capture all the panic messages. – kzs Jan 02 '14 at 12:50
  • @kzs, `cat /proc/kmsg` hangs, waiting for the kernel to say something. I think it provides messages from the live kernel, not from the one that has crashed. Also `klogd` polls it and saves the data. – ugoren Jan 02 '14 at 13:12
  • otherwise try either of the two options(preferably 2nd): 1.`cat /proc/kmsg > //mylog.txt &` (as your system boots- let it run in background). 2.add the above to init script so that it stores logs to your mylog.txt from booting to kernel crash and rename it to some name(next boot does not disturb previous log). -- Later on next boot you can read the panic messages from the previous logs. – kzs Jan 03 '14 at 05:59
  • @kzs, I'm pretty sure `klogd` does pretty much this. But it doesn't always catch crash messages. – ugoren Jan 03 '14 at 16:53

1 Answers1

1

The kernel module 'netconsole' may help you to log kernel printk messages over UDP.

You can view the log message in remote syslog server, event if the machine is rebooted.

Introduction:
=============

This module logs kernel printk messages over UDP allowing debugging of
problem where disk logging fails and serial consoles are impractical.

It can be used either built-in or as a module. As a built-in,
netconsole initializes immediately after NIC cards and will bring up
the specified interface as soon as possible. While this doesn't allow
capture of early kernel panics, it does capture most of the boot
process.

Check kernel document for more information: https://www.kernel.org/doc/Documentation/networking/netconsole.txt

Brightshine
  • 975
  • 1
  • 7
  • 17
  • Thanks, but this requires an extra machine to collect the logs, and I can't rely on having one. I want to have the information on the same machine, after it reboots. – ugoren Jan 02 '14 at 12:03