python multiprocessing handling an array of numbers concurrently

Question

I have a list of number:

a=[1,2,3,4,5,.....2000]

I have to square each number and update the same array, but instead of writing a loop i want to do it using parallel processing.

So squaring each number in the array becomes a process in itself.

Expected output=[1,3,9,16,25,........]

How can i achieve this with python multiprocessing library?

Already tried to Use threading library but the code is not fast enough, plus Threading library is not using all the cores.

If you are able/willing to use `numpy` then you could do this a lot quicker than I imagine multiprocessing ever could. — roganjosh, Jul 28 '17 at 06:15
Also for your last point, threading cannot increase the speed of this task. Due to the Global Interpreter Lock (GIL), only one thread can execute its code at any one time. Threading only gives the illusion of concurrency. — roganjosh, Jul 28 '17 at 06:17
Multiple processes are not bound by the GIL, only multithreading is @roganjosh — 101, Jul 28 '17 at 06:44
@101 I never said multiprocessing was bound by the GIL (although each subprocess is). My comment specifically says threading, to address the last sentence for the question... — roganjosh, Jul 28 '17 at 06:48
numpy only let's you perform numerical calucations right?...i cant do some pattern matching over a string on arrays or something similar..? — ATIF ADIB, Jul 28 '17 at 08:07

Akshay Apte · Answer 1 · 2017-07-28T06:10:42.547

2

You can use Pool class from the multiprocessing module

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    p = Pool(5)
    print(p.map(f, [1, 2, 3]))

#prints [1, 4, 9]

edited Jul 28 '17 at 06:10

answered Jul 28 '17 at 06:08

Akshay Apte

1,539
9
24

1

Pool(5) indicating there are 5 cores in the cpu?? – ATIF ADIB Jul 28 '17 at 06:10
I got a very weird error: TypeError: cannot serialize '_io.TextIOWrapper' object ...i am reading the array from a json file... – ATIF ADIB Jul 28 '17 at 07:20
Possible solution for that: https://stackoverflow.com/questions/26249442/can-i-use-multiprocessing-pool-in-a-method-of-a-class – Akshay Apte Jul 28 '17 at 07:29
No wait i solved it my question is: def f(x,power): return pow(x,power) how do i pass power to the function f? – ATIF ADIB Jul 28 '17 at 07:45

score 0 · Answer 2 · answered Jul 28 '17 at 06:50

In this numpy would be handy because it works on Matrix methods to calculate. Here is the piece of code that can serve the purpose. In case you want to parallel it you can use the Pool function as stated

import numpy as np
def Square(data):
    data_np = np.array(data) ** 2
    print (data_np)
Square([1, 2, 3])

score 0 · Answer 3 · answered Jul 28 '17 at 08:58

You can try ProcessPoolExecutor in concurrent.futures module. Example code:

from time import time
from concurrent.futures import ProcessPoolExecutor


def gcd(pair):
    a, b = pair
    low = min(a, b)
    for i in range(low, 0, -1):
        if a % i == 0 and b % i == 0:
            return i


numbers = [(1963309, 2265973), (2030677, 3814172),
           (1551645,    2229620),   (2039045,   2020802)]
start = time()
results = list(map(gcd, numbers))
end = time()
print('1st Took %.3f seconds' % (end - start))
start = time()
pool = ProcessPoolExecutor(max_workers=2)
results = list(pool.map(gcd, numbers))
end = time()
print('2nd Took %.3f seconds' % (end - start))

python multiprocessing handling an array of numbers concurrently

3 Answers3