1

When I initialize as p = Process(target=fn), where fn is some function call.

Is there a way to specify, say an argument to the Process initialization such that the maximum number of cores allocated to p when it runs is bounded by a given number?

I could not find such a param in the documentation here.

piedpiper
  • 1,222
  • 3
  • 14
  • 27
  • 1
    https://stackoverflow.com/questions/20886565/using-multiprocessing-process-with-a-maximum-number-of-simultaneous-processes – Matthew Kligerman Jan 03 '22 at 03:29
  • I'm asking a different question I believe.. – piedpiper Jan 03 '22 at 03:32
  • you may find if you can get your function(s) to happily work with a `multiprocessing.Pool` that you're better off, and it will give you shared runtime to see if you should do more optimization work, especially if you have a problem of the form of multiple functions which don't need each others' results, but you need the collective output from to continue further – ti7 Jan 03 '22 at 03:46

1 Answers1

2

It's up to your function to determine how many cores it uses (or rather is able to take advantage of), not your Process instance

I think this is something of an inversion to how you're designing, but processes are not locked to any or even some set of cores other than how the operating system assigns them, generally providing them time on the resources it manages (CPU, memory, network, disk..), which they practically request via its API.

Unless your process goes out of its way to take advantage of multiple cores, it will only be able to take advantage of at most one core (thread) and likely less overall work than the core possibly can.
However, many 3rd party libraries (such as NumPy) will take advantage of more cores unless otherwise instructed by creating additional threads, normally as many as the system supports by default. If this is the case, you can adjust the number of threads they use, usually via arguments or environmental variables.

Take a look at these resources

Once created, you could set operating system-level restrictions (cgroup settings for most Linux-like systems) on your new process to change how it's scheduled if the parent process has sufficient permissions to do so, but this may result in worse performance than expected (for example, if you create 8 threads, but restrict usage to 2 cores, overhead time will be wasted switching between threads that are not able to get more value)

See also

ti7
  • 16,375
  • 6
  • 40
  • 68