9

I use web.py to create a Python web server. This server is called to solve linear programming problems, and it uses the library CBC to do that.

Every once in a while, the server crashes with a log that looks like that:

78.243.184.3:56271 - - [03/Jun/2016 04:35:54] "HTTP/1.1 GET /optimization" - 200 OK
Aborted (core dumped)

I belive "Aborted (core dumped)" is a C error, so it comes from either web.py or CBC.

Is there any way to trace back the source of the error?

Arnaud
  • 4,884
  • 17
  • 54
  • 85

2 Answers2

6

A core dump is caused by a fault in the native code in your web server. Python is pretty darn solid these days, so such faults are almost always caused by bugs in C extensions in my experience.

You therefore have 3 problems.

  1. You need to find the core dump file. This is usually in the current working directory of the process that dumped. However, there are configation options that can change this.

  2. You need to debug the call stack of what has failed. That is covered in StackOverflow - see How to analyze a program's core dump file with gdb?

  3. You might need to relate the Python interpreter stack back to the Python code that you were running. To do this you will need to install the Python debug symbols and Python extensions for gdb. The Python wiki has good advice on how to do that here.

Community
  • 1
  • 1
Peter Brittain
  • 13,489
  • 3
  • 41
  • 57
  • Thanks for this answer, I will check gdb tomorrow. I have several questions please: 1. how do you know the problem comes from the web server and not CBC? 2. do you acknowledge that the problem comes from the library web.py and not from my code? 3. I don't find any core dump file in the directory of server.py. Where can I find the configuration options of web.py? 4. How to debug it if it is in the code of web.py? 5. Do you think the best option would be to change web.py for another library? Thanks for your help! – Arnaud Jun 08 '16 at 22:09
  • 1
    I geninely can't tell you the answers. You have a core dump and running gdb on that will tell you where the bug is. That is why core files exist - for post mortem analysis. In order to get more help from anyone, you need to extract the call stack using the techniques I have described and update your question with it. – Peter Brittain Jun 08 '16 at 22:19
  • But the bug appears juste every once in a while (once every 2 or 3 weeks), and I am not able to reproduce it... – Arnaud Jun 09 '16 at 17:38
  • 1
    I'm really sorry, but there is nothing more I can do here. Core files exist precisely to diagnose these sorts of failures. Find the core file, do the stack trace and then share your findings if that hasn't pinpointed the bug for you. All those previous dumps will have generated files. You can use any of them. – Peter Brittain Jun 09 '16 at 18:49
2

Most common reason for a Core dump is to access a memory address outside of your reach (memory that do not belong to your program). The operatingsystem interrupts the program with a interruption called SegFault or BusError depeding on how the program tried to access the invalid memory address. The program will then be forceful closed down by the kernel. If the kernel has been configured to create a core-dump (a dump of memory and stack of the program) one will be saved to disc. As noted in the other answer you may load the core-dump into GDB or other debugger and display what the program was doing at the time it crashed. This might or might not give you the problem at hand. Normally it's hard even for a experienced programmer to used these tools so be aware. If you want to try to use GDB, try this:

$ gdb /path/to/crashing/program/binary /path/to/core

(gdb) bt

'bt' will display 'backtraces' otherwise known as StackTraces, and can be useful for a programmer to track down the error.

If you are able to reproduce the error, you are probably lucker submitting a detailed bugreport to the creator of the program in question. Even me as a senior software developer takes that route from time to time. :-)

UnixShadow
  • 1,222
  • 8
  • 12