1

i wrote the following code, that runs a command using subprocess method, under 2 for loops, 

import threading
import subprocess

def do (i,j):
    subprocess.Popen(['python2.7 some_python_code.py {} {}'.format(i,j)],shell=True,stdout=subprocess.PIPE)

z=[]
for i in range (0,50):
    for j in range (0,50):
        t = threading.Thread(target=do, args=(i,j))
        t.start()
        z.append(t)

for t in z:
    t.join()

the matter is that it suddenly starts a massive number of python process and eats the entire of my memory, how can i limit the number of threads ?

thanks.

anon
  • 433
  • 1
  • 8
  • 14
  • 1
    (1) `subprocess.Popen()` returns immediately while the process is running yet. You must `wait` for it to terminate. (2) For the thread creation look at `ThreadPoolExecutor` to restrict the number of threads. – Michael Butscher Oct 10 '17 at 00:59
  • i think subprocess.call waits for the result but it's serial process then ,not parallel process. thanks – anon Oct 10 '17 at 04:50

2 Answers2

1

Maybe if you use the option setDaemon you can improve the issue, I will put here a link to the reference, read the section Daemon vs noDaemon. https://pymotw.com/2/threading/

By default this option is not activated you can activate in the following way:

daemon_thread = threading.Thread(name='MyDaemon', target=do_some_stuff)
daemon_thread.setDaemon(True)

You can check the following link if the first option doesn't resolve the problem How to reduce memory usage of threaded python code? I hope that the link can help you, good luck!

Jose Moreno
  • 104
  • 4
  • thanks alot Jose, the matter is that it runs 50*50 process at the same time . – anon Oct 10 '17 at 04:48
  • Then you need use multiprocessing, thread don't execute jobs in a parallel way, if you want parallel jobs you need used multiprocessing – Jose Moreno Sep 24 '20 at 03:44
1

You should be using concurrent.futures - that automatically creates worker pools of the desired size to which tasks as submited in order.

Also, you have to change your subprocess call to be blocking - if the do function returns immediately, using the pool won't do a thing.

import concurrent.futures

import subprocess
import sys

def do (i,j):
    subprocess.Popen(['python2.7 some_python_code.py {} {}'.format(i,j)],shell=True,stdout=subprocess.PIPE)

with concurrent.futures.ThreadPoolExecutor(max_workers=8) as pool:
    for i in range (0,50):
        for j in range (0,50):
            t = pool.submit(do, i,j)

   for task in concurrent.future.as_completed(pool):
        try:
            result = task.result()
        except Exception as error:
            print(f"Error executing task '{task}': {error}", file=sys.stderr)
jsbueno
  • 99,910
  • 10
  • 151
  • 209