How to launch 100 workers in multiprocessing?

Question

I am trying to use python to call my function, my_function() 100 times. Since my_function takes a while to run, I want to parallelize this process.

I tried reading the docs for https://docs.python.org/3/library/multiprocessing.html but could not find an easy example to get started with launching 100 workers. Order does not matter; I just need the function to run 100 times.

Any suggestions/code tips?

Use a Pool of size 100. This question gives plenty of examples. https://stackoverflow.com/questions/44660676/python-using-multiprocessing — David Parks, Jan 07 '22 at 19:55
BTW a typical pool size is a little more than the number of available CPUs. So I guess you have an impressive machine. — Klaus D., Jan 07 '22 at 20:05
My machine is google colab. So what would be the fastest way to run 100 times? — Galen BlueTalon, Jan 07 '22 at 20:10
Simon not necessarily. If the function e.g. waits on IO (especially on HTTP requests or something, but also on files etc), then there's not much harm in having 100 threads. Sure, *usually* there's a more elegant way of multiplexing such things, but sometimes there isn't. — Marcus Müller, Jan 07 '22 at 20:10
How many cores do you get from Colab? It's not that many. 1 or 2? — Marcus Müller, Jan 07 '22 at 20:10
Thanks for your response regarding Colab. Do you have any suggestions as to the fastest way to run my function 100 times, on colab? — Galen BlueTalon, Jan 07 '22 at 20:17
Actually I forgot to mention that my_function actually uses the GPU. — Galen BlueTalon, Jan 07 '22 at 20:17
With threads I am fine, but processes is a different thing in my view. Multithreading for Io bound problems wäre you spin up many AS you are just waiting and Multiprocessing for CPU bound problems where you only create as many processes as you have cores. — Simon Hawe, Jan 07 '22 at 20:24
@GalenBlueTalon if you want to run a GPU function one hundred times in parallel, you're completely off. multithreading helps you **not at all** here. You need to start your computation with the hundred datasets using the framework you use for your interaction with the GPU. You forgot to say the most important thing! I'd recommend you **ask a new question** specifically about **accellerating your GPU computation** and you show **your actual code** in that new question post. This is very frustrating, honestly! — Marcus Müller, Jan 07 '22 at 21:37

Marcus Müller · Accepted Answer · 2022-01-07T20:14:11.927

The literally first example on the page you link to works. So I'm just going to copy and paste it here and change two values.

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(100) as p:
        print(p.map(f, range(100)))

EDIT: you just said that you're using Google colab. I think google colab offers you two cpu cores, not more. (you can check by running !cat /proc/cpuinfo). With 2 cpu cores, you can only execute two pieces of computation at once.

So, if your function is not primarily something that waits for external IO (e.g. from network), then this makes no sense: you've got 50 executions competing for one core. The magic of modern multiprocessing is that this means that suddenly, one function will be interrupted, its state saved to RAM, the other function then may run for a while, gets interrupted, and so on.

This whole exchanging of state of course is overhead. You'd be faster just running as many instances your function in parallel as you have cores. Read the documentation on Pool as used above for more information.

How to launch 100 workers in multiprocessing?

1 Answers1

Linked