Android Kernel Debugging

Question

I have been experimenting with getting kgdb to work the Nexus One.

I have pulled the kernel from https://android.googlesource.com and enabled everything to do with kgdb including kgdbts testing using menuconfig. Successfully built the kernel and flashed it to the device (which is unlocked rooted and running CyanogenMod 7)

I have also followed the instructions found on http://bootloader.wikidot.com/android:kgdb to enable the usb connection to act as a serial connection as required by kgdb (and tested communications from ttyACM0 to ttyGS0 successfully).

The following folders exist indicating that kgdboc and kgdbts have been built into the kernel:

/sys/modules/kgdboc/parameters
/sys/modules/kgdbts/parameters

The following is the output from dmesg showing the kgdbts testing being done show what (I think) is a successful completion of the tests:

# dmesg | grep kgdb
<6>[   12.974060] kgdb: Registered I/O driver kgdbts.
<6>[   12.981781] kgdbts:RUN plant and detach test
<6>[   12.995178] kgdbts:RUN sw breakpoint test
<6>[   13.002441] kgdbts:RUN bad memory access test
<6>[   13.010864] kgdbts:RUN singlestep test 1000 iterations
<6>[   13.019042] kgdbts:RUN singlestep [0/1000]
<6>[   13.077850] kgdbts:RUN singlestep [100/1000]
<6>[   13.132720] kgdbts:RUN singlestep [200/1000]
<6>[   13.187500] kgdbts:RUN singlestep [300/1000]
<6>[   13.242370] kgdbts:RUN singlestep [400/1000]
<6>[   13.297149] kgdbts:RUN singlestep [500/1000]
<6>[   13.351928] kgdbts:RUN singlestep [600/1000]
<6>[   13.406829] kgdbts:RUN singlestep [700/1000]
<6>[   13.461578] kgdbts:RUN singlestep [800/1000]
<6>[   13.516540] kgdbts:RUN singlestep [900/1000]
<6>[   13.570922] kgdbts:RUN do_fork for 100 breakpoints
<6>[   21.117645] kgdb: Unregistered I/O driver kgdbts, debugger disabled.

I believe that the problem I am running into is getting the kernel to trigger kgdb.

# echo -n g > /proc/sysrq-trigger

Just results in dropping me back to the command prompt and (I think) it is suppose to freeze everything and send a prompt over usb which is being used as a psuedo serial port since the phone doesn't have a real one.

From what I gather from my research that prompt is supposed to be the trigger that will allow me to issue

(gdb) target remote /dev/ttyACM0

And connect to a debugging session with the kernel.

I have also tested /proc/sysrq-trigger with b and c just confirm that I am able to pass some commands to sysrq.

So my question following my long winded attempt to provide as much information as possible is why is g not triggering the debugger?

This is my first attempt at kernel debugging on any system and I have run out of ways to phrase my search on google so I turn to you. Thanks!

(I have also tried putting kdgbwait in the kernel command line without success as I believe this is not yet supported by the android kernel)

Answer 1

Android Kernel questions are rare on [SO], asa nobody else has answered I have provided my findings on this issue. Unfortunately I don't have a nexus one to test this on so this answer is not intended as a step by step resolution of you problem, but should point you in the right direction of where to look.

The only useful resource I have found on this problem is in a LKML patch by Dongdong Deng, so it is unlikely a configuration issue as these are usually abundant and well-publicized.

This indicates that there is a problem with your kernel build. I would be tempted to start again with the latest versions of CM and see if the problem just goes away.

Failing that, try reporting this to the cyanogen team and see if this is a known issue or has a simple workaround.

As a last resort you could try the patch if the versions are compatible. The only alternative is to roll up your sleeves and start hacking the CM kernel to incorporate the patch.

Good luck.

Answer 2

Found this post from a related post and wanted to say I've just published some work I did to get this working on the Nexus 6 if anyone is interested:

http://www.contextis.com/resources/blog/kgdb-android-debugging-kernel-boss/

Interestingly, the OP's problem with sysrq was one I also came across. The reason for this behaviour is that KGDB wasn't initialising correctly so it fails to install the handler for the 'g' (kgdb) trigger. That's why all the other sysrq commands still work.

Longer explanation (thanks @Robert):

To get this working I had to make a UART debug cable based on this Accuvant blog. This is quite a simple circuit which consists of a FTDI 3.3v basic breakout (available from SparkFun at the time of writing), as well as 4 resistors (2 x 1K Ohm, 1 x 1.2K Ohm and 1 x 100Ohm), and a 4-element Tip-Ring-Ring-Sleeve (TRRS) headphone jack. The resistors are essentially providing a voltage divider to reduce the 3.3v down to something a little safer for your phone. By inserting the audio jack with the other end connected to your circuit board, the audio subsystem recognises that a voltage (~2.8V) on the one of the pins and it knows to provide a UART interface via that cable. The FTDI breakout plugs into your PC via USB and from here you can access console messages via a terminal emulator like minicom. However, you now have a serial interface through the same mechanism and that's what we can use for a KGDB connection.

So at this point some relatively minor changes are required to the Nexus 6's serial driver (msm_serial_hs_lite.c) to support KGDB (specifically, the ability to perform atomic character I/O operations). I just ported these changes from the Linux Kernel mainline code as a chap called Stephen Boyd had done the hard work to the full MSM (Qualcomm) serial driver msm_serial.c. His changes can be found here or just search for "msm_serial: add support for poll_" on Google. The port wasn't difficult and my code can be found on github.

Aside from that you need to be able to build a custom kernel for your N6 which google provides lots of information on. You then need to create a boot image which contains the KGDB modifications in the github repo. I took the stock kernel from https://developers.google.com/android/nexus/images, extracted it (using abootimg -x) and then used the following command to repack it with my custom kernel (zImage-dtb) and additional command line params to ensure KGDB would be loaded and point to my serial port like so:

abootimg -u boot.img -k zImage-dtb -c 'cmdline=console=ttyHSL0,115200,n8 kgdboc=ttyHSL0,115200 kgdbretry=4'

With my boot.img created I could boot into it using the command fastboot boot boot.img, open an adb shell and then trigger a breakpoint in the Android kernel using the command:

echo -n g > /proc/sysrq-trigger

It is worth mentioning for completeness that you need superuser privileges to access /proc/sysrq-trigger so you need to have root.

With the phone halted, and your debug cable connected, launch a version of GDB for ARM on your host PC with your uncompressed kernel as an argument (e.g. arm-eabi-gdb ./vmlinux). Note: I'm running Ubuntu 14.04 and using arm-eabi-gdb from the 'prebuilts' directory in my AOSP source repository. Finally, enter the following commands:

set remoteflow off
set remotebaud 115200
target remote /dev/ttyUSB0

All being well this should immediately break into the kgdb breakpoint (that your write to /proc/sysrq-trigger produced) and you can start debugging.

Answer 3

I have got no experience with Android hardware, but I have done kgdb-compiled kernel running as VirtualBox client, and from the host connect into the guest via virtual serial port, and using gdb (with standard "target remote" command) I can step through the entire bootup of the virtual guest kernel - with the help of kgdbwait. Without this, I can write a kernel module that does nothing except to implement an inline assembly that called "int 13", which is 0xcc. Once loaded, a breakpoint will appear at the host-side of the serial connection, and then I can set breakpoint and continue excution of the kernel. This works because kgdb handle the exception "int 13". If u explicitly create other kind of exception like "*p = 0", and p is pointing to NULL, u will still get a breakpoint, but I doubt if u can continue execution.