I think I have once read about this but can't find it now.
I'm running linux-5.4.188 on a qemu arm64 virtual machine. Because I built the kernel from the source, I can debug(analyze) the kernel by attaching to the linux kernel program running on a remote machine(qemu virtual machine). To test an application which uses our device(the device model is in qemu too), I compiled a device driver against kernel 5.4.188 and the linux application and can do insmod the driver and run the application.
Now something is wrong and I have panic while running the application. I can debug linux kernel itself, but I don't know where the kernel module was loaded, so the debugger cannot debug the driver module. How can I debug the device driver? (or even the application? in case I need to someday). I remember by first getting the loaded address of the kernel module, and doing add-symbol-file for the driver image relative to that loaded address, it is possible to do kernel module debug. I think this is what driver developers will be doing always. Please tell me how I can do it. If this is possible, it will save many days for me.

- 5,177
- 12
- 57
- 112
-
I generally just use `printk` to narrow down the problem. – Ian Abbott Apr 05 '22 at 10:38
-
See https://www.kernel.org/doc/html/latest/dev-tools/gdb-kernel-debugging.html – stark Apr 05 '22 at 12:46
-
Thanks for the comments, I found how to do it and put an answer for others' reference. – Chan Kim Apr 05 '22 at 14:01
1 Answers
As Ian Abbott said I use printk for kernel debugging. Usually for this I put his this line in /etc/sysctl.conf. (for ubuntu-20.04 case. I'm not sure it's applicable to vanila linux too)
kernel.printk = 5 4 1 7
The first value is the console log level and the second value is default log level. By making default 4 (WARNING) and console log level 5 (NOTICE), every default printk appears in the console (because 5 is higher thatn 4, less value is more important and more important messages than the console log level gets printed in the console. see How can I show printk() message in console?). This way you don't have to check with dmesg everytime. You can see the kernel __log_buf before the serial port is initialized in setup_arch function in start_kernel. The kernel message are written in memory called __log_buf before serial port is initialized.
I refered to https://www.kernel.org/doc/html/latest/dev-tools/gdb-kernel-debugging.html that stark pointed me to and I also found https://wiki.st.com/stm32mpu/wiki/Debugging_the_Linux_kernel_using_the_GDB very useful.
Important things during the kernel config :
set CONFIG_GDB_SCRIPTS=y, CONFIG_DEBUG_INFO_REDUCED=n, CONFIG_FRAME_POINTER=y.
I use nokaslr in the kernel command line, but added CONFIG_RANDOMIZE_BASE=n to make sure.
Now the part I originally wanted to know from my question :
In the shell (in virtual machine linux shell) do,
$ls -la /sys/module/<module_name>/sections
Then you'll see files like .text, .data, .bss, etc. You cd to that directory and do
$cat .text .data .bss
To see the address of each section. In my case,
/sys/module/axpu_ldd_kc/sections # cat .bss .data .text
0xffff800008ca5480
0xffff800008ca5000
0xffff800008ca0000
and in the gdb, (after stopping the program by ctrl-c) I did
add-symbol-file ~/testlin540/axpu_ldd_kc.ko 0xffff800008ca0000 -s .data 0xffff800008ca5000 -s .bss 0xffff800008ca5480
and I tried setting breakpoint at my ioctl function.
(gdb) b axpu_ioctl
and when I pressed c (continue) and when I started my application, I could see the program stop at the axpu_ioctl and I could single step through the code and see the values.
When I did kernel debug for booting using u-boot recently, I frequently wrapped a function with
#pragma GCC push_options
#pragma GCC optimize ("O0")
and
#pragma GCC pop_options
To prevent some parts of the codes from being optimized away. (sometime you should do it for the related #include <xxx.h> statement too to prevent compile error).

- 5,177
- 12
- 57
- 112