17

I am writing a program that grades student programs, and as I am sure you can imagine, they sometimes segmentation fault. The problem I am having is that when the student programs segmentation fault, there is no indication that is what happened.

proc = subprocess.Popen(student_command, 
                        stdout=subprocess.PIPE, 
                        stderr=subprocess.PIPE)
self.stdout, self.stderr = proc.communicate()
self.returncode = proc.returncode

I pick up the stderr, stdout, and the return code from the subprocess, but if the program segmentation faults, stderr is empty, stdout is empty, and the return code is -11. Now I could look for the -11 exit code and assume that if that is the return code there was a segmentation fault, but there is also nothing to prevent a student's code from having -11 as a return code just because the student felt like returning -11.

How do you tell if a subprocess segmentation faults, as opposed to just feeling like returning -11? I don't really care all that much about what is in stderr and stdout, and to that end have seen a number of posts including this that deal with picking up the output, but I don't care all that much about the output, although it would be nice to get the "Segmentation Fault" string out of stderr, but what I really need is a way to definitively tell what happened to the subprocess.

Neuron
  • 5,141
  • 5
  • 38
  • 59
zelinka
  • 3,271
  • 6
  • 29
  • 42
  • Do note that student code can also die in other marvelous ways, like `SIGABRT` (managing to trigger an assert by e.g. trashing the heap), `SIGBUS` (misaligned access on some platforms), `SIGXCPU` (by running out of CPU time, assuming you have the appropriate limiters in place), `SIGFPU` (divide by zero), and all sorts of other gruesome ways. Be prepared for abuse. – nneonneo Sep 11 '13 at 03:00
  • Also, instead of hard-coding "-11", you should use the symbolic constants in the `signal` module. – nneonneo Sep 11 '13 at 03:01

1 Answers1

10

Well, in fact, on UNIX, a process that attempts to return -11 will usually end up returning a positive integer instead. This is because the return status from the wait series of functions is actually a set of bitfields, with a field for the signal that ended the process and a separate field for the return value. Python decodes the wait return value from these bitfields.

On most systems, these fields are unsigned and 8 bits in size, so you will probably see something like this:

>>> import subprocess
>>> subprocess.Popen(['python','-c','import os; os.kill(os.getpid(),11)']).wait()
-11
>>> subprocess.Popen(['python','-c','exit(-11)']).wait()
245

In the former case, the process "segfaults" (by killing itself with SIGSEGV), and so wait returns -11. In the latter case, the process exits with a return code of -11, and the resulting wait value is 245 (256-11). You can therefore rest assured that any negative return value from wait must represent a fatal signal, as opposed to a normal return. Note, though, that processes may kill themselves to fake a fatal error.

nneonneo
  • 171,345
  • 36
  • 312
  • 383
  • So it is possible to differentiate between a program that just returned -11 and a process that died from a segfault by checking the sign of the return value? – zelinka Sep 13 '13 at 16:26
  • Why is it not returning -11 in the latter? – JobHunter69 Nov 08 '19 at 21:58
  • On most UNIXy systems, including Linux, return codes are unsigned and 8 bits in size, so -11 gets transformed to 245 (the unsigned 8-bit twos-complement representation of -11). – nneonneo Nov 08 '19 at 22:20