Measuring runtime of CUDA through an script in python

Question

I am trying to use some profiling tools on CUDA with so many different inputs to see the the differences among different inputs. For this purpose I wrote a python script to compile the code on CUDA, run the profiler with different inputs and write the results into a CSV file. Some of these runs take so much time ( about 3 days ! ) and I want to kill that specific run if it takes more than a threshold ( for example 30 minutes ) but I do no know how to do it. Here is a part of python script that I wrote :

import subprocess ,sys, string, os
{START TIME}
p = subprocess.Popen([CUDA PROFILER COMMAND], stdout=subprocess.PIPE)
s, err = p.communicate()
{END TIME}

I want to measure START TIME and END TIME and kill this run if it takes more than 30 minutes. The script would run line by line and if the CUDA PROFILER command takes more than 30 minutes, it never gets to {END TIME} line and I cannot measure the elapsed time of the CUDA PROFILER command. Any suggestion would be so much appreciated.

As a matter of fact I think I found the answer in another question. Here is the link for all of those who have the same question : http://stackoverflow.com/questions/1191374/subprocess-with-timeout?lq=1 — user3540236, Apr 16 '14 at 08:41
if you found an answer; you could [post your own answer](http://stackoverflow.com/help/self-answer) — jfs, Apr 16 '14 at 12:51
related: [Stop reading process output in Python without hang?](http://stackoverflow.com/q/4417962/4279) — jfs, Apr 16 '14 at 13:15

score 1 · Answer 1 · edited May 23 '17 at 10:25

There are many answers in subprocess with timeout question that you've linked in the comments. I've chosen a couple that are both simple and portable. In Python 3.3+:

from subprocess import check_output

s = check_output(['cuda', 'profiler', 'command'], timeout=30*60)

On older versions:

from subprocess import Popen, PIPE
from threading import Timer

p = Popen(['cuda', 'profiler', 'command'], stdout=PIPE)
t = Timer(30*60, p.kill) # kill process in 30 minutes
t.start()
s = p.communicate()[0] 
t.cancel() # cancel the hit

Measuring runtime of CUDA through an script in python

1 Answers1