4

I am writing a python script to automate debugging core dump from gdb. i am trying to print data structure which includes kernel data structures and lists(e.g. struct list_head). For example the structure is something like this:

struct my_struct {
  struct my_hardware_context ahw;
  struct net_device *netdev;
  struct pci_dev *pdev;
  struct list_head mac_list;
  ....
  ....
};

i am using following API tp print this structure:

gdb.execute('p (*(struct my_struct *)dev_base->priv)')

so i am able to print the content of 'struct my_struct' , struct my_hardware_context ahw, but not the content of pointers and list ( e.g. struct net_device *netdev, struct pci_dev *pdev, struct list_head mac_list) automatically (only address is printed). So how to print the content of *netdev, *pdev and mac_list using gdb-python script?

EDITED : to make my question more clear

I am writing a python script to automate debugging core dump from gdb. i am trying to print data structure which includes kernel data structures and lists(e.g. struct list_head). For example the structure is something like this:

struct my_struct {
   struct my_hardware_context ahw;
   struct net_device *netdev;
   struct pci_dev *pdev;
   struct list_head mac_list;
   ....
   ....
};

i am using following API to print this structure: (it can be assumed that i have right core dump and added proper symbols.

main_struct = gdb.execute('p (*(struct my_struct *)dev_base->priv)')

print main_struct

Now it will print the values of all members of struct my_struct but upto one level , meaning it will print the whole content of struct my_hardware_context ahw because it is an instance but it will not print the content of struct net_device *netdev, struct pci_dev *pdev, struct list_head mac_list etc. so now manually i need to do it like below:

netdev = gdb.parse_and_eval('*(*(struct my_struct *)dev_base->next->priv).netdev')

print netdev

pdev = gdb.parse_and_eval('*(*(struct my_struct *)dev_base->next->priv).pdev')

print pdev

so i want to automate these steps. Is there any gdb-python API or way by which it can iterate the struct my_struct and print the pointers, arrays and lists values also automatically?

Thanks.

Baijnath Jaiswal
  • 377
  • 5
  • 17

1 Answers1

10

struct net_device, struct pci_dev from Linux are meant to be used by kernel and not userspace code. They're not even exported in the sanitized kernel headers you get with make headers_install for use with libc.

GDB can't print struct net_device, struct pci_dev because it doesn't have debug info describing the definition of those structures. Your userspace struct my_struct is declared to have opaque pointers to those structures. I don't think you should be doing that in the first place.

Edit After Core Dump Clarification

The trick is loading debug info from both the kernel and your driver module into GDB:

  • Grab a kernel with debuginfo (CONFIG_DEBUG_INFO). e.g. for Centos, get the matching kernel-debuginfo package from http://debuginfo.centos.org/6/x86_64/.
  • Get the .text, .data and .bss load addresses of your driver module by inspecting /sys/module/MY-DRIVER/sections/{.text,.data,.bss} from a system running your driver under normal operation.

Assuming the kernel with debug info is located at /usr/lib/debug/lib/modules/3.9.4-200.fc18.x86_64/vmlinux, run:

$ gdb /usr/lib/debug/lib/modules/3.9.4-200.fc18.x86_64/vmlinux vmcore
(gdb) add-symbol-file MY-DRIVER.ko TEXT-ADDR -s .data DATA-ADDR -s .bss BSS-ADDR

while replacing TEXT-ADDR, DATA-ADDR and BSS-ADDR with the address from the files under /sys/module/MY-DRIVER/sections/. (I think just lying and using an address of 0 would probably work in this case)

Verify that ptype struct net_device, ptype struct pci_dev, ptype my_struct work. Then after obtaining the address of a struct *my_struct the way you did before you should be able print its contents.

Traversing a Struct While Following Pointers

print-struct-follow-pointers.py

import gdb

def is_container(v):
    c = v.type.code
    return (c == gdb.TYPE_CODE_STRUCT or c == gdb.TYPE_CODE_UNION)

def is_pointer(v):
    return (v.type.code == gdb.TYPE_CODE_PTR)

def print_struct_follow_pointers(s, level_limit = 3, level = 0):
    indent = ' ' * level

    if not is_container(s):
        gdb.write('%s\n' % (s,))
        return

    if level >= level_limit:
        gdb.write('%s { ... },\n' % (s.type,))
        return

    gdb.write('%s {\n' % (s.type,))
    for k in s.type.keys():
        v = s[k]
        if is_pointer(v):
            gdb.write('%s %s: %s' % (indent, k, v))
            try:
                v1 = v.dereference()
                v1.fetch_lazy()
            except gdb.error:
                gdb.write(',\n')
                continue
            else:
                gdb.write(' -> ')
            print_struct_follow_pointers(v1, level_limit, level + 1)
        elif is_container(v):
            gdb.write('%s %s: ' % (indent, k))
            print_struct_follow_pointers(v, level_limit, level + 1)
        else:
            gdb.write('%s %s: %s,\n' % (indent, k, v))
    gdb.write('%s},\n' % (indent,))

class PrintStructFollowPointers(gdb.Command):
    '''
    print-struct-follow-pointers [/LEVEL_LIMIT] STRUCT-VALUE
    '''
    def __init__(self): 
        super(PrintStructFollowPointers, self).__init__(
            'print-struct-follow-pointers',
            gdb.COMMAND_DATA, gdb.COMPLETE_SYMBOL, False)

    def invoke(self, arg, from_tty):
        s = arg.find('/')
        if s == -1:
            (expr, limit) = (arg, 3)
        else:
            if arg[:s].strip():
                (expr, limit) = (arg, 3)
            else:
                i = s + 1
                for (i, c) in enumerate(arg[s+1:], s + 1):
                    if not c.isdigit():
                        break
                end = i
                digits = arg[s+1:end]
                try:
                    limit = int(digits)
                except ValueError:
                    raise gdb.GdbError(PrintStructFollowPointers.__doc__)
                (expr, limit) = (arg[end:], limit)
        try:
            v = gdb.parse_and_eval(expr)
        except gdb.error, e:
            raise gdb.GdbError(e.message)

        print_struct_follow_pointers(v, limit)

PrintStructFollowPointers()

Sample Session

(gdb) source print-struct-follow-pointers.py
(gdb) print-struct-follow-pointers *p

You can limit the levels of embedded structures printed:

(gdb) print-struct-follow-pointers/4 *p
scottt
  • 7,008
  • 27
  • 37
  • So do you want to change your answer to "how are you generating the coredump?" ;) – scottt May 28 '13 at 10:08
  • yes, Actually it is generated automatically when the driver crashes into directory /var/core – Baijnath Jaiswal May 28 '13 at 10:15
  • is it possible in this way, by redirecting output of `(gdb) ptype struct my_struct` into a file and parse each field(symbol) which is pointer,array or list and then get the type of that symbol. finally `(gdb) p *` ? – Baijnath Jaiswal May 28 '13 at 11:22
  • Thanks scottt. I have done all the steps you mentioned, meant added the proper symbols(picked corresponding addresses from kernel log file) and also verified that `gdb.execute('p *(*(struct my_struct *)dev_base->priv).netdev')` is working properly but there are so many struct pointers like netdev and pdev. so i want to get all these pointers and arrays automatically and print their contents. you can assume that i have added proper symbols of driver and kernel. – Baijnath Jaiswal May 28 '13 at 11:43
  • @BaijnathJaiswal, I've amended my answer. – scottt May 28 '13 at 12:45
  • Thanks scottt:-) i have restricted it to upto 2 level. I am moving further..if i face any issue, i will let you know. – Baijnath Jaiswal May 28 '13 at 14:18
  • 1
    I've modified the code again to add a `-l` option as well. This code is really just using `gdb.Value` and `gdb.Type` in the obvious way to do the recursive traversal. Just read those sections in the GDB manual. The required Python knowledge, how to define functions, classes, how to handle exceptions etc can be picked up in an hour by reading the official language tutorial :) – scottt May 28 '13 at 14:49
  • i was wrong, i am not able to restrict the loop at level 2, and when it comes to `struct net_device *netdev`, it is in infinite loop. :-( it is not reachable to print `struct pci_dev *pdev` and further. – Baijnath Jaiswal May 28 '13 at 14:49
  • Try the version currently in the answer. – scottt May 28 '13 at 14:57
  • it is not taking argument. i am providing this `print-struct-follow-pointers (*(struct my_struct *)dev_base->priv) -l 3` , showing usage error message `print-struct-follow-pointers [-l LEVEL_LIMIT] STRUCT-VALUE.` – Baijnath Jaiswal May 28 '13 at 15:00
  • I've changed the limit option syntax in the code. `print-struct-follow-pointers/4 EXPR` should work now. – scottt May 28 '13 at 15:30