How to debug python core dump?

Question

My python program calls functions inside a shared object. There are two core dump files. But I can't get any useful information from them. My python program calls functions inside a shared object. There are two core dump files. But I can't get any useful information from them.

(gdb) bt
#0  0x00007f9a436b6d44 in PyObject_stgdict () from /home/ubuntu/bingo/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
#1  0x00007f9a436b66e5 in _ctypes_callproc () from /home/ubuntu/bingo/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
#2  0x00007f9a436b7f9e in ?? () from /home/ubuntu/bingo/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
#3  0x0000000000499be5 in PyEval_EvalFrameEx ()
#4  0x0000000000499ef2 in PyEval_EvalFrameEx ()
#5  0x00000000004a090c in PyEval_EvalCodeEx ()
#6  0x000000000049ab45 in PyEval_EvalFrameEx ()
#7  0x00000000004a090c in PyEval_EvalCodeEx ()
#8  0x000000000049ab45 in PyEval_EvalFrameEx ()
#9  0x00000000004a1c9a in ?? ()
#10 0x0000000000505f96 in PyObject_Call ()
#11 0x000000000049b07a in PyEval_EvalFrameEx ()
#12 0x0000000000499ef2 in PyEval_EvalFrameEx ()
#13 0x0000000000499ef2 in PyEval_EvalFrameEx ()
#14 0x0000000000499ef2 in PyEval_EvalFrameEx ()
#15 0x0000000000499ef2 in PyEval_EvalFrameEx ()
#16 0x0000000000499ef2 in PyEval_EvalFrameEx ()
#17 0x00000000004a090c in PyEval_EvalCodeEx ()
#18 0x000000000049ab45 in PyEval_EvalFrameEx ()
#19 0x00000000004a090c in PyEval_EvalCodeEx ()
#20 0x000000000049ab45 in PyEval_EvalFrameEx ()
#21 0x0000000000499ef2 in PyEval_EvalFrameEx ()
#22 0x00000000004a1c9a in ?? ()
#23 0x00000000004dfe94 in ?? ()
#24 0x00000000004dc9cb in PyEval_CallObjectWithKeywords ()
#25 0x0000000000588b12 in ?? ()
#26 0x00007f9a449de184 in start_thread (arg=0x7f99f1ffb700) at pthread_create.c:312
#27 0x00007f9a4470b03d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

(gdb) bt
#0  0x000000000054eee0 in ?? ()
#1  0x000000000057392b in ?? ()
#2  0x0000000000536476 in ?? ()
#3  0x0000000000537827 in _PyObject_GC_New ()
#4  0x000000000052fcb2 in PyList_New ()
#5  0x00000000005ae57d in ?? ()
#6  0x000000000051eb00 in ?? ()
#7  0x000000000049a3b5 in PyEval_EvalFrameEx ()
#8  0x00000000004a090c in PyEval_EvalCodeEx ()
#9  0x000000000049ab45 in PyEval_EvalFrameEx ()
#10 0x00000000004a090c in PyEval_EvalCodeEx ()
#11 0x000000000049ab45 in PyEval_EvalFrameEx ()
#12 0x00000000004a090c in PyEval_EvalCodeEx ()
#13 0x000000000049ab45 in PyEval_EvalFrameEx ()
#14 0x0000000000499ef2 in PyEval_EvalFrameEx ()
#15 0x0000000000499ef2 in PyEval_EvalFrameEx ()
#16 0x00000000004a090c in PyEval_EvalCodeEx ()
#17 0x000000000049ab45 in PyEval_EvalFrameEx ()
#18 0x00000000004a090c in PyEval_EvalCodeEx ()
#19 0x000000000049ab45 in PyEval_EvalFrameEx ()
#20 0x0000000000499ef2 in PyEval_EvalFrameEx ()
#21 0x00000000004a1c9a in ?? ()
#22 0x00000000004dfe94 in ?? ()
#23 0x00000000004dc9cb in PyEval_CallObjectWithKeywords ()
#24 0x0000000000588b12 in ?? ()
#25 0x00007fad2dbdb184 in start_thread (arg=0x7fad255d3700) at pthread_create.c:312
#26 0x00007fad2d90803d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

score 1 · Accepted Answer · answered Jun 04 '21 at 14:41

I can't get any useful information from them.

On the contrary: you did get useful info from them.

Both crashes are inside Python, and one of them is somewhere inside allocation routines. This is usually a sign of heap corruption.

My python program calls functions inside a shared object.

If by "shared object" you mean a library that you (or someone else) developed, it's possible that this library has a heap corruption bug. Such bugs are nasty in that their effect (e.g. crash) often happens 1000s of instructions later, and they are notoriously hard to find without specialized tools.

Luckily we do have such tools: Valgrind and Address Sanitizer.

I would start by running your program under Valgrind. Note that you need a special Python build for this to be effective.

How to debug python core dump?

1 Answers1