Python Run function in parallel

Question

Is there a way to speed this up by running in parallel? The longest process time is scipy.ndimage.map_coordinates.

import multiprocessing
import numpy as np
import scipy.ndimage

pool = multiprocessing.Pool()
n=6
x0=350
y0=350
r=150
num=10000
#z = np.gradient(sensor_dat, axis=1)
z = np.random.randn(700,700)

def func1(i):
    x1, y1 = x0 + r * np.cos(2 * np.pi * i / n), y0 + r * np.sin(2 * np.pi * i / n)
    x, y = np.linspace(x0, x1, num), np.linspace(y0, y1, num)
    zi = scipy.ndimage.map_coordinates(z, np.vstack((y, x)))
    return zi

[func4(i) for i in range(36)]
#pool.map(func1,range(36))

I tried from Is there a simple process-based parallel map for python? to use pool.map(func1,range(36)) but got error Can't pickle <function func1 at 0x0000019408E6F438>: attribute lookup func1 on __main__ failed

I found How to accelerate scipy.map_coordinates for multiple interpolations? but dont think this is relevent as scipy.ndimage.map_coordinates is the majority of process time, but dont think it will speed up in my case.

score 0 · Accepted Answer · answered Apr 07 '22 at 22:27

0

Yes you can. Just follow the instruction within the multiprocessing documentation and measure whether it will actually be faster using multiple workers.

Here is the code that I tested with:

from multiprocessing import Pool
import numpy as np
from scipy import ndimage
from time import time

n=6
x0=350
y0=350
r=150
num=10000
#z = np.gradient(sensor_dat, axis=1)
z = np.random.randn(700,700)

def f(i):
    x1, y1 = x0 + r * np.cos(2 * np.pi * i / n), y0 + r * np.sin(2 * np.pi * i / n)
    x, y = np.linspace(x0, x1, num), np.linspace(y0, y1, num)
    zi = ndimage.map_coordinates(z, np.vstack((y, x)))
    return zi

if __name__ == '__main__':
    begin = time()
    [f(i) for i in range(36)]
    end = time()
    print('Single worker took {:.3f} secs'.format(end - begin))

    begin = time()
    with Pool() as p:
        p.map(f, list(range(36)))
    end = time()
    print('Parallel workers took {:.3f} secs'.format(end - begin))

This yields the following output on my machine:

Single worker took 0.793 secs
Parallel workers took 0.217 secs

answered Apr 07 '22 at 22:27

Dudly01

444
3
13

Thanks for some reason I had to use `from multiprocessing.pool import ThreadPool as Pool`. I will see if Pool works when I move it over to Linux. Thanks for help – ADDonut Apr 07 '22 at 22:39
1

@MTR You might want to know, Python Interpreter since ever WAS & WILL (as Guido van ROSSUM, the Father of Python Interpreter, said himself) for any foreseeable future, most probably forever remain a CONCURRENT-execution principally avoided ecosystem (all Python threads have to first "borrow" a MUTEX GIL-lock, before they can do some small amount of work ~ 100 [ms])-so any and all concurrency is principally avoided (prevented) by this,as Guido wanted it to be so. Use of thread-based pool only increase costs (add-on overheads) while processing remains a split ALL-wait-1-works a pure-[SERIAL] ... – user3666197 Apr 08 '22 at 07:02
@MTR, just as user3666197 said, if you wanna get your job done in parallel, then multiprocessing is the way to go. (At least as far as my insights go.) Due to the Global Interpreter Lock (GIL), threads are not working in parallel to each other as just one of them are working at a specific moment in time. But whatever you do, do not forget to also measure the execution time! – Dudly01 Apr 08 '22 at 18:31
Much appreciate with all the help and info! – ADDonut Apr 10 '22 at 18:34

Python Run function in parallel

1 Answers1