what tool for debugging a linux kernel?

Question

I am new to linux kernel. wandering how to browse the complete flow, right from the power up of CPU. Basic idea on BIOS/ROM code.

can I have some tool to debug the complete kernel ? or raw code browsing is preferable ?

Answer 1

The following tools may help you to debug Linux kernel

Dynamic Probes is one of the popular debugging tool for Linux which developed by IBM. This tool allows the placement of a “probe” at almost any place in the system, in both user and kernel space. The probe consists of some code (written in a specialized, stack-oriented language) that is executed when control hits the given point. Resources regarding dprobes / kprobes listed below

• http://www-01.ibm.com/support/knowledgecenter/linuxonibm/liaax/dprobesltt.pdf
• http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.107.6212&rep=rep1&type=pdf
• https://www.redhat.com/magazine/005mar05/features/kprobes/
• https://sourceware.org/systemtap/kprobes/
• http://www.ibm.com/developerworks/library/l-kprobes/index.html
• https://doc.opensuse.org/documentation/html/openSUSE_121/opensuse-tuning/cha.tuning.kprobes.html

Linux Trace Toolkit is a kernel patch and a set of related utilities that allow the tracing of events in the kernel. The trace includes timing information and can create a reasonably complete picture of what happened over a given period of time. Resources of LTT, LTT Viewer and LTT Next Generation

• http://elinux.org/Linux_Trace_Toolkit
• http://www.linuxjournal.com/article/3829
• http://multivax.blogspot.com/2010/11/introduction-to-linux-tracing-toolkit.html

MEMWATCH is an open source memory error detection tool. It works by defining MEMWATCH in gcc statement and by adding a header file to our code. Through this we can track memory leaks and memory corruptions. Resources regarding MEMWATCH

• http://www.linuxjournal.com/article/6059

ftrace is a good tracing framework for Linux kernel. ftrace traces internal operations of the kernel. This tool included in the Linux kernel in 2.6.27. With its various tracer plugins, ftrace can be targeted at different static tracepoints, such as scheduling events, interrupts, memory-mapped I/O, CPU power state transitions, and operations related to file systems and virtualization. Also, dynamic tracking of kernel function calls is available, optionally restrictable to a subset of functions by using globs, and with the possibility to generate call graphs and provide stack usage. You can find a good tutorial of ftrace at https://events.linuxfoundation.org/slides/2010/linuxcon_japan/linuxcon_jp2010_rostedt.pdf

ltrace is a debugging utility in Linux, used to display the calls a user space application makes to shared libraries. This tool can be used to trace any dynamic library function call. It intercepts and records the dynamic library calls which are called by the executed process and the signals which are received by that process. It can also intercept and print the system calls executed by the program.

• http://www.ellexus.com/getting-started-with-ltrace-how-does-it-do-that/?doing_wp_cron=1425295977.1327838897705078125000
• http://developerblog.redhat.com/2014/07/10/ltrace-for-rhel-6-and-7/

KDB is the in-kernel debugger of the Linux kernel. KDB follows simplistic shell-style interface. We can use it to inspect memory, registers, process lists, dmesg, and even set breakpoints to stop in a certain location. Through KDB we can set breakpoints and execute some basic kernel run control (Although KDB is not source level debugger). Several handy resources regarding KDB

• http://www.drdobbs.com/open-source/linux-kernel-debugging/184406318
• http://elinux.org/KDB
• http://dev.man-online.org/man1/kdb/
• https://www.kernel.org/pub/linux/kernel/people/jwessel/kdb/usingKDB.html

KGDB is intended to be used as a source level debugger for the Linux kernel. It is used along with gdb to debug a Linux kernel. Two machines are required for using kgdb. One of these machines is a development machine and the other is the target machine. The kernel to be debugged runs on the target machine. The expectation is that gdb can be used to "break in" to the kernel to inspect memory, variables and look through call stack information similar to the way an application developer would use gdb to debug an application. It is possible to place breakpoints in kernel code and perform some limited execution stepping. Several handy resources regarding KGDB

• http://landley.net/kdocs/Documentation/DocBook/xhtml-nochunks/kgdb.html

Answer 2

First, see related question Linux kernel live debugging, how it's done and what tools are used?. Try to use KDB or Ftrace.

Answer 3

If your intention is understanding whole flow of Linux kernel, running Linux kernel on QEMU can be easy way to learn how Linux works. Esp. you can emulate many CPU types without real H/W. or how about user mode Linux?

This document can be helpful to debug kernel on QEMU.

Answer 4

Just adding, the Linux kernel is not very suitable for debugging. Linus Torvalds once stated that he's againts supportng kernel debugging in Linux because it leads to badly written code. I used kdbg, however I didn't find it very useful, what I suggest is to debug the kernel the oldschool way, using printk.