1

I am trying to test some programs by using Python. I want to see if given certain input they crash, end without errors, or run for longer than a timeout.

Ideally I would like to use subprocess as I am familiar with that. However am able to use any other useful library. I assume that reading core dump notifications is an option, however I do not yet know how to do that nor do I know if that is the most effective way.

HanMah
  • 129
  • 3
  • 2
    Can you share the code that you have tried – Jeril Apr 05 '19 at 05:11
  • I'm not sure how to answer your request. I don't know what code to try. I read the subprocess manual and did not see any way to see if the program ended in an error. – HanMah Apr 05 '19 at 05:20
  • You can start by calling a python script using subprocess and raise an error / exit the called python script. – Jeril Apr 05 '19 at 05:22
  • you can use logging, flags or any notify packages to get the status of the running script. – Arun Augustine Apr 05 '19 at 05:28
  • You may want to take a look at https://stackoverflow.com/questions/636561 and https://stackoverflow.com/questions/1191374. You could start a sub-process in a separate thread with a timeout. Checking the exit code is pretty straightforward once it's done. – kingkupps Apr 05 '19 at 05:28
  • @Jeril , Sure, I did that. I don't see how that helps as I don't know what to read from that. I did print(subprocess.call("")) . Can you be more blunt as to what you are suggesting? I don't understand your hints. – HanMah Apr 05 '19 at 05:31
  • @ArunAugustine I don't see how polling would help as a program can end either with no error (but return a non-zero value) or by causing through eg. a segfault – HanMah Apr 05 '19 at 05:32
  • @kingkupps see my response to ArunAugstine – HanMah Apr 05 '19 at 05:33

1 Answers1

0

Using os and What is the return value of os.system() in Python?, a solution could be:

status = os.system(cmd)
# status is a 16 bit number, which first 8 bits from left(lsb) talks about signal used by os to close the command, Next 8 bits talks about return code of command.
sig, ret = os.WIFSIGNALED(status), os.WEXITSTATUS(status)
# then check some usual problems:
if sig:
    if status == 11:      # SIGSEGV
        print ('crashed by segfault')
    elif status == 6 :    # SIGABRT
        print('was aborted')
    else: # 14, 9 are related to timeouts if you like them
        print('was stopped abnormally with', status)
else:
    print('program finished properly')

I haven't checked yet if subprocess returns the same status.

Demi-Lune
  • 1,868
  • 2
  • 15
  • 26
  • Thank you for your answer. It appears to do what I require. For some reason I get status 139 when running a program that ends in SIGSEV . Any idea why? – HanMah Apr 05 '19 at 07:23
  • A program that ends in SIGFPE causes status 136. I guess that there is an offset – HanMah Apr 05 '19 at 07:33
  • The offset is 128, that's a convention when a program is stopped by a signal (linux or bash convention I don't remember; it is probably explained on SO somewhere). I must have over-simplified my code snippet... – Demi-Lune Apr 05 '19 at 16:32
  • Upon further testing I realize that if the program ends in an exception, I can tell which one it is by using the table here:https://www.tutorialspoint.com/unix/unix-signals-traps.htm However, if the program finished without an exception, but returned a non-zero value, then I cannot tell the difference between that and a program that crashed. – HanMah Apr 10 '19 at 13:09