1

I'm a long time SO reader, but this is my first question here.
I'm writing a Python 3 program (which I'm currently self-learning) to compile and run a group of C++ programs that supposed to be doing the same thing (students' submissions for an assignment) and compare their output to the expected output I already have in a text file.

I'm looking for a way to run these executables that allows me to:

  • Terminate an executable after a certain time limit (say, 10 seconds) if it doesn't end.
  • Capture all the output from the executable (including stdout, stderr, and any "shell" messages like segmentation faults, memory/core dump,..etc)
  • Even if the executable crashes, or has been forcefully terminated (after exceeding the allowed time), I need to capture the "partial" output up to that point.
  • Nothing goes to the screen (No output or error messages).

Everything I have tried so far (from answers to similar questions) is missing something. I couldn't find one that covers all my needs (the points above). Here are some examples:

  • check_output() (like in this question) does not collect the output if the executable crashes, and some messages (like core dump) still go to the screen.

  • Popen() (as in this) does not seem to terminate the executable (when time exceeded). The python thread ends (I hope I'm using the correct terms here), but the executable keeps running.

  • Even though methods like os.system() are not encouraged, but I tried them anyway, and they didn't perform any better.

I appreciate it a lot if any one can point me to a way to achieve this. I apologize for any grammatical mistakes as English is not my first language. Please let me know if any further details are needed. Thank you,

EDIT:

Here is a minimum replica of my python code:

from subprocess import STDOUT, check_output, TimeoutExpired

timeLimit = 3  # seconds
strOutput = ''

try:
    output = check_output('./a.out', stderr = STDOUT, timeout = timeLimit)
    strOutput = ''.join(map(chr, output))

except TimeoutExpired as e:
    strOutput += 'Error: Time limit exceeded, terminated..'

except Exception as e:
    strOutput += 'Error: ' + str(e)

f = open('report.txt','w')
f.write(strOutput)
f.close()

I'm also including some C++ samples that contain some bugs to create the executables to be used to test the previous code. They all can be compiled as: g++ programName.cpp

#include <iostream>
using namespace std;

int main()
{
   cout << "This one is working correctly!" << endl;
}

The following code causes a "division by zero" error. The statement printed before this error is not captured by check_output()

#include <iostream>
using namespace std;

int main()
{
   cout << "Trying to divide by zero.." << endl;
   cout << 3 / 0 << endl;
}

This code waits for an unexpected input (should be terminated by the script after the specified timeout)

#include <iostream>
using namespace std;

int main()
{
   int x;
   cout << "Waiting for an unexpected input.." << endl;
   cin >> x;
}

I couldn't find a reliable way to reproduce the core dump issue. Core dump is a special case issue because check_output() cannot capture its output (it goes directly/only to the screen).

P.S. I couldn't add the tag 'check_output()', I do not have enough reputation.

Elman
  • 11
  • 2
  • 1
    What OS are you using? – unutbu Oct 08 '17 at 22:18
  • I'm currently running on my Linux Mint 18.1 laptop, but this script is intended to run on the school's Ubuntu 16.04 server. – Elman Oct 08 '17 at 22:27
  • `subprocess.call()` with `timeout` calls `p.kill()` and `p.wait()` in its exception handler, which is invoked should that timeout expire. As such, it *does* in fact kill the immediately-spawned process. – Charles Duffy Oct 09 '17 at 00:50
  • Now, if the process you spawned has children, killing it may not kill the children -- but if that's your problem, you should provide a [mcve] letting others reproduce it, as the best-practice approach for a fix will vary depending on the details. In some cases, the right fix may be to modify the way you invoke it to flatten the process tree, thus avoiding having those children at all. (If you have an intermediate shell script, having it `exec` the program it spawns is within this category of solutions). – Charles Duffy Oct 09 '17 at 00:51
  • Thank you all for your replies.. I edited the question to include the required code sections. – Elman Oct 09 '17 at 04:15

2 Answers2

0

A simple solution could be to use all the solutions you have found so far, but instead of running the programs once, run them multiple times.

one time, to determine running time, another time, to determine output etc. this way, you can also test if given the same input they provide the same output. it may not be the "super elegant" solution you are looking for, but it may suffice for now until you find the holy grail of subprocess in python.

also, you can redirect the output to a file in order to have partial results+timeout and then read the file for the partial results.

Felipe Valdes
  • 1,998
  • 15
  • 26
  • Thank you for the answer. The running time is not actually an issue. I'm using it only to avoid programs with endless loops. My main issue is to capture *all* the output. I couldn't find a way to redirect the output to a file using check_output(), so I appreciate it if you can point me to a working example (I'm relatively new to python). – Elman Oct 09 '17 at 04:34
  • Here is what I was trying (to redirect to a file) `output = check_output(['./a.out', ' >report.txt'], stderr = STDOUT, timeout = timeLimit)` – Elman Oct 09 '17 at 04:44
  • In addition to compiling the programs, I also do some basic analysis on the compiler's output (to check for certain warnings, count specific errors,..etc). I also do not expect the output to be an **exact** match to what I have already (so using something like _diff_ is not an option). I decided to go with a "higher" level language like python because I don't know how to do all that with bash. – Elman Oct 09 '17 at 14:36
  • Perhaps the problem can be trying to grade a program with errors and warnings. if it doesn't terminate correctly, or if it doesn't do what it's supposed to, what is the point?, would industry clients would accept half baked software?. to me, either they run and do the thing, or they don't. – Felipe Valdes Oct 09 '17 at 16:43
0

Based on @CharlesDuffy comments, I simplified the process tree to make it flatter, and switched to subprocess.call() which has the advantage of capturing "partial" output (before the executable crashes).
Core dump, however, is still not captured by subprocess.call(), it goes to the screen.
Is there any way to capture the output of core dump? or at least to prevent it from showing on the screen?

from subprocess import call, TimeoutExpired

timeLimit = 3  # seconds
errMsg = ''
ret_code = 0
f = open('report.txt','w')

try:
    ret_code = call('./a.out', stdout = f, stderr = f, timeout = timeLimit)

except TimeoutExpired as e:
    errMsg = 'Error: Time limit exceeded, terminating..'

except Exception as e:
    errMsg = 'Error: ' + str(e)

if ret_code:
    errMsg = 'Error: ' + str(ret_code)

f.write(errMsg)
f.close()
Elman
  • 11
  • 2