-1

I have found the following lines of code to compute the array mx by the repeated calling of a function called fun.

However, I would like to understand better what it does.

Also, I assigned 16 cores to the parallel pool, however, I noticed that during computations no more than 2 cores are running at the same time.

Could someone explain what this code does and why it could be that only part of the threads is working?

Thank you!

from tqdm import tqdm
from multiprocessing import Pool
from functools import partial

with Pool(processes = 16) as p_mx: 
    mx = tqdm(p_mx.imap(partial(fun, L), nodes), total = n)
Adam Gosztolai
  • 242
  • 1
  • 3
  • 14
  • 2
    well, let's start by how many CPU cores do you have? Note that these are *processes*, not threads. – Paritosh Singh Jul 25 '19 at 09:23
  • Do you even have 16 cores? As you can see, `Pool` literally calls it "processes", not cores. Number of actual cores it gets executed on depends on how many cores you have and how your resources get assigned by your OS. – h4z3 Jul 25 '19 at 09:23
  • Thanks, the computer has 2 x 22 cores. When I run the script, most of them are on idle. – Adam Gosztolai Jul 25 '19 at 09:27
  • "Could someone explain what this code does" It does *lots* of things, from calling ``imap`` on a ``multiprocessing.Pool`` down to managing all sorts of structs with your kernel. What *exactly* do you need clarification on? – MisterMiyagi Jul 25 '19 at 10:27
  • How do you check the number of cores used? How long does the computation take? How long does each individual call take? What is ``fun``, ``L`` and ``nodes``? Why don't you set a ``chunksize`` above the default of ``1``? – MisterMiyagi Jul 25 '19 at 10:36

1 Answers1

0

multiprocessing.Pool() slower than just using ordinary functions

The function you are trying to parallelize doesn't require enough CPU resources (i.e. CPU time) to rationalize parallelization!

And may caused by the way Python handle multi-threading and multi-processing with the GIL:

When to use threading and how many threads to use

Look at the GIL, you will have a better understanding of why.

If you want concurrent code in Python 3.8, you have CPU-bound concurrency problems then this could be the ticket!

Dorian Turba
  • 3,260
  • 3
  • 23
  • 67