Restrict number of subprocess.Popen

Question

Following from run multiple instances of python script simultaneously I can now write a python program to run multiple instances.

import sys
import subprocess

for i in range(1000):
  subprocess.Popen([sys.executable, 'task.py', '{}in.csv'.format(i), '{}out.csv'.format(i)])

This starts 1000 subprocess simultaneously. If the command that each subprocess is runs is computationally resource intensive, this can result in load on the machine (may even crash).

Is there a way where I can restrict the number of subprocess to be run at a time? For example something like this:

if (#subprocess_currently_running = 10) {
   wait(); // Or sleep 
}

That is just allow 10 subprocess to run at a time. In case one out of ten finishes start a new one.

Answer 1

Counting Semaphore is an old-good mechanism which can be used to control/manage the maximum number of concurrently running threads/processes.

But as each subprocess.Popen object (implying process) needs to be waited for termination, the official doc tells us about the important downside of subprocess.Popen.wait()(for this case of multiple concurrent sub-processes):

Note: The function is implemented using a busy loop (non-blocking call and short sleeps). Use the asyncio module for an asynchronous wait: see asyncio.create_subprocess_exec.

Thus, it's preferable for us to switch to:

• asyncio.create_subprocess_exec
• asyncio.Semaphore

---

How it can be implemented:

import asyncio
import sys

MAX_PROCESSES = 10


async def process_csv(i, sem):
    async with sem:  # controls/allows running 10 concurrent subprocesses at a time
        proc = await asyncio.create_subprocess_exec(sys.executable, 'task.py',
                                                    f'{i}in.csv', f'{i}out.csv')
        await proc.wait()


async def main():
    sem = asyncio.Semaphore(MAX_PROCESSES)
    await asyncio.gather(*[process_csv(i, sem) for i in range(1000)])


asyncio.run(main())