0

I have script that contains heavy CPU-bound function that loops through list and i want it to be using multiprocessing.

script looks like

... do stuff
    
function_result = my_heavy_function()

... do stuff after i get function_result

After reading some about multithreading and multiprocessing i tried to do

import concurrent.futures

... do stuff
    
with concurrent.futures.ProcessPoolExecutor() as executor:
    executor.map(my_heavy_function, my_list)

... do stuff after i get function_result

But it took ~minute longer than default version of my script which uses no parallel things. (idfk why, bless you if you can explain it to me)

So all i need is to get result of that function asap and then work with this result in 'normal tempo'. Is it possible to use multiprocessing in this way or am i doing something wrong?

MLReio
  • 21
  • 2
  • Not sure exactly what your heavy function looks like, but you might try breaking up the list into chunks and pass those to workers as seen here: https://stackoverflow.com/questions/8717179/chunking-data-from-a-large-file-for-multiprocessing. I think the `ProcessPoolExecutor.map` may just be running your function multiple times in parallel...https://stackoverflow.com/questions/41903230/understanding-concurrent-futures-executor-map#42192177 – Sparrow1029 May 27 '21 at 11:38
  • i thought ProcessPoolExecutor does the same as multiprocessing.Pool.map which is as you say breaks list in chunks (https://docs.python.org/3/library/multiprocessing.html) – MLReio May 27 '21 at 12:01
  • Hmmm, you're right. What about some of the techniques in this thread/answer? https://stackoverflow.com/a/55754990/6768111 Seems they were using a `Manager()` on the pool of workers which has some overhead, but the answer I linked might be performant. Maybe it would help if you posted what your function is actually doing, as that may be relevant? – Sparrow1029 May 28 '21 at 00:52
  • @Sparrow1029 My function takes list as an argument and does some heavy math with its elements. As i can understand from the docs, doesn't matter what exactly function does, main idea is to split list into chunks and work with them in different threads/processes instead of iterating over whole thing at once. And That's what i'm trying to do. That thread https://stackoverflow.com/questions/55754501/multiprocessing-on-chunks-of-an-image/55754990#55754990 is not relevant to my problem but thank you anyway) – MLReio May 28 '21 at 07:22

0 Answers0