4

I want to run a serial program on multiple cores at the same time and I need to do that multiple time (in a loop). I use subprocess.Popen to distribute the jobs on the processors by limiting the number of jobs to the number of available processors. I add the jobs to a list and then I check with poll() if the jobs are done, I remove them from the list and continue the submission until the total number of jobs are completed.

I have been looking on the web and found a couple of interesting scripts to do that and came out with my adapted version:

nextProc = 0
processes = []
while (len(processes) < limitProc):     # Here I assume that limitProc < ncores
  input = filelist[nextProc]+'.in'      # filelist: list of input file
  output = filelist[nextProc]+'.out'    # list of output file
  cwd = pathlist[nextProc]              # list of paths
  processes.append(subprocess.Popen(['myProgram','-i',input,'-screen',output],cwd=cwd,bufsize=-1))
  nextProc += 1
  time.sleep(wait)

while (len(processes) > 0):                     # Loop until all processes are done
  time.sleep(wait)
  for i in xrange(len(processes)-1, -1, -1):    # Remove processes done (traverse backward) 
    if processes[i].poll() is not None:
      del processes[i]
    time.sleep(wait) 
  while (len(processes) < limitProc) and (nextProc < maxProcesses):    # Submit new processes
    output = filelist[nextProc]+'.out'
    input = filelist[nextProc]+'.in'
    cwd = pathlist[nextProc]
    processes.append(subprocess.Popen(['myProgram','-i',input,'-screen',output],cwd=cwd,bufsize=-1))
    nextProc += 1
    time.sleep(wait)

print 'Jobs Done'

I run this script in a loop and the problem is that the execution time increases from one step to another. Here is the graph: http://i62.tinypic.com/2lk8f41.png

myProgram time execution is constant. I'd be so glad if someone could explain me what is causing this leak.

Thanks a lot, Begbi

Begbi
  • 148
  • 1
  • 5
  • consider using `multiprocessing.Pool`. It sets one process per CPU, runs jobs, and starts new jobs when old ones are completed. http://stackoverflow.com/questions/11436502/closing-all-threads-with-a-keyboard-interrupt – johntellsall Jun 27 '14 at 16:25
  • 1
    I tried with multiprocessing and have similar problems. I'd rather understand where the leak is coming from more than an alternate solution. – Begbi Jul 03 '14 at 17:40

0 Answers0