0

I'm planning a list of subprocesses of some.exe e.g.

max_processes = 4
proc_list = []
for i in range(100):
    while len(proc_list) <= max_processes:
        proc_list.append(subprocess.Popen(["some.exe", i]))

The processes should work in the background and the number of max_processes should constantly run. The std out of some.exe is not required.

This is a rough plan of what I'd like to do, is this even possible or do I have to rethink the whole concept?

Daedalus Mythos
  • 565
  • 2
  • 8
  • 24
  • 1
    related: [Python threading multiple bash subprocesses?](http://stackoverflow.com/q/14533458/4279) – jfs Sep 30 '14 at 17:04

1 Answers1

2

You can use multiprocessing.Pool or multiprocessing.pool.ThreadPool.

from multiprocessing import Pool
# OR  from multiprocessing.pool import ThreadPool as Pool
import subprocess

def run_some_exe(i):
    subprocess.call(['some.exe', i])

if __name__ == '__main__':
    max_processes = 4
    pool = Pool(max_processes)
    pool.map(run_some_exe, range(100))

mulitprocess.Pool.map works like map except that it run the function parallelly. (max 4 in above example)

falsetru
  • 357,413
  • 63
  • 732
  • 636
  • thank you very much for the suggestions, how would I accomplish the `pool.map(lambda [...]` if I'd have to determine the `i` on another way - say `if some_var = False: i = some_list[j]`. Am I correct to say that I could wrap a `while some_bool:` around the `pool.map[...]` and assign `i` each time and call `pool.map(subprocess.call(['some.exe', i]))`. Would i have *some.exe* running 4 times, all the time - until *some_bool* becomes False? – Daedalus Mythos Sep 30 '14 at 13:51
  • 1
    Note that this isn't going to work on Windows, because of the use of a lambda inside the `if __name__ == "__main__"` guard. You'll need to use a top-level function outside the guard, instead. – dano Sep 30 '14 at 13:52
  • @dano, Thank you for the comment. I updated the answer accordingly. – falsetru Sep 30 '14 at 13:53
  • 1
    @DaedalusMythos, I updated the answer to use a function instead of a lambda. For your case, I would make a shared list for all processes, and use it in the `run_some_exe` function. (BTW, note that I used `subprocess.call` instead of `subprocess.Popen` to make each process wait the termination of the `some.exe`) – falsetru Sep 30 '14 at 13:56
  • 3
    One other note: It actually might make more sense to use `multiprocessing.pool.ThreadPool` for this, since we don't actually need the separate processes to avoid the GIL. – dano Sep 30 '14 at 13:57
  • @dano, updated the answer again to add that. Thank you. – falsetru Sep 30 '14 at 14:00
  • thanks a lot for the update, looks almost like what I had in mind. If i issue `pool.map(run_some_exe)` in a `while a_boolean:`, where i declare `i` in the run_some_exe, will it keep 4 processes up until `a_boolean` is false? – Daedalus Mythos Sep 30 '14 at 14:00
  • @DaedalusMythos, The works to do is not fixed beforehand? like `range(100)` in `pool.map(run_some_exe, range(100))` – falsetru Sep 30 '14 at 14:01
  • @falsetru the `i` which I put simply as `i in range(100)` is actually a value in a database, which is queried before assigned to a *some.exe*.. – Daedalus Mythos Sep 30 '14 at 14:04
  • 2
    @DaedalusMythos, Then, replace the `range(100)` with the list of the values fetched. – falsetru Sep 30 '14 at 14:05
  • @falsetru of course can I use the list there, but unfortunately do I have to set a bit in the database for each successfully processed `i` by *some.exe* (which might crash). You pointed me in the right direction - which helps a lot :) thank you – Daedalus Mythos Sep 30 '14 at 14:11
  • 1
    @DaedalusMythos, You have a function `run_some_exe`. Do whatever you want there :) – falsetru Sep 30 '14 at 14:12
  • haha you're right.. But still can I `while a_boolean: pool.map(run_some_exe)` to keep the 4 *threads/processes/whatever* running until `a_boolean` is False? – Daedalus Mythos Sep 30 '14 at 14:30
  • 1
    @DaedalusMythos, Yes you can do it. But it will create 4 processes (threads) inside the loop; and wait for them to finish jobs. – falsetru Sep 30 '14 at 14:32
  • Will it wait as soon as I call the `run_some_exe` function (e.g. `run_some_exe` - wait until it's finished, then start the next `run_some_exe`) or will it create 4, wait until *one* is finished and then start a new one - repeating until above mentioned boolean is False? – Daedalus Mythos Sep 30 '14 at 14:44
  • 1
    @DaedalusMythos, the latter. – falsetru Sep 30 '14 at 14:51
  • 1
    you could use a thread pool instead of process pool. Change import to `multiprocessing.dummy`. – jfs Sep 30 '14 at 17:06