How to assign subpool (m workers) of multiprocessing pool (n workers with m < n) to some task in python?

Question

I have usecase like following: I have total 20 multiprocessing workers. I can give all resources to task1. However, task2 has lower priority and I can at most give it half of total resources. How to assign subpool (m workers) of multiprocessing pool (n workers with m < n) to some task in python? Or is there some design pattern to handle this usecase?

import multiprocessing

pool = multiprocessing.get_context("fork").Pool(20)


def task1():
    # want to use all 20 workers
    pass


def task2():
    # only want to use 10 workers
    pass

PS: For example, I have 20 physical cores, and my tasks are all CPU-bound task (every task can only use one core, since it's pure python code without any c++ package), therefore I hope at most 20 running workers at same time.

Task1 and task2 are incoming async jobs with number n and m (n >> 20 and m >> 20). In our requirement, task2 cannot consume half of total cores at same time. However, task1 can use all resources.

You could have task2 temporarily lower its process priority. This [old answer](https://stackoverflow.com/a/1023269/642070) seems like it would still work, I think. Its not strictly 10 and 20 workers, but fiddling with priority can be a useful way to handle these things. — tdelaney, Dec 19 '22 at 20:48

Aaron · Answer 1 · 2022-12-19T22:49:54.420

You can always create multiple pools:

ctx = multiprocessing.get_context("fork")
with ( #Parenthesized context managers new in 3.10 !!!
    ctx.Pool(20) as p1,
    ctx.Pool(10) as p2
):
    r1 = p1.imap_async(task1, args1)
    r2 = p2.imap_async(task2, args2)
    for a, b in zip(r1, r2):
        print(a, b)

Idle workers won't consume much CPU, so it's not a bad idea to slightly over-provision each sub-pool. They do consume some memory however, so keep that in mind.

Edit Regarding running more processes than cores:

Your computer has hundreds of processes going at any time (on most modern operating systems) it is the job of the operating system scheduler to prioritize each one based on how active they are. If you have more processes than you have physical cores, the OS will be actively (and rapidly) swapping between them to make sure each one runs at least once in a while (probably on the order of miliseconds). The process of switching is fast but not completely free, so it does hurt performance to create many many more processes than you have cores.

If you have enough work for all 30 workers to remain busy, and they are all assigned the same priority by the OS (they will unless you change it), they will roughly split the cpu time 2:1 as each process should get roughly equal time. If you run out of task2 to work on, the cpu will stay fully busy with 20 workers, though if you run out of task1, you'll only use 50% of your cores. I've used strategies like this in the past, and it's simple and functional. In reality, if you need to squeeze every last drop of performance from your cpu, you'll need to do a lot of testing and heuristic based tuning for a method like this to work(which will need to be constantly updated any time any code or data changes).

If you do need something more complicated, you could create your own "Pool" from Process, Queue, Lock, etc.. with the exact number of workers as you have cores, and manually set priority (eg; pulling tasks for the children from a priority queue).

Thanks. In your example, it will at most 30 workers run at same time, right? I have total 20 physical cores, therefore I hope at most 20 workers at same time. For example, 1000 task1 and 1000 task2. I hope tasks2 "at most" consume 10 physical cores at same time. — maplemaple, Dec 19 '22 at 22:23

How to assign subpool (m workers) of multiprocessing pool (n workers with m < n) to some task in python?

1 Answers1