Multiprocessing a function's output to a single array

Question

I have been trying this for quite sometime now, but my array remains unchanged.

My array here is TC_p_value, and the function I am trying to simulate is TC_stats. The code runs fine if we run it normally, but takes too long to simulate (about an hour). Thus, to reduce the processing time, I divided the original array (1000x100) in 10 small sets of 100x100. Although, the code runs without an error, I somehow always get the same array (same as it is defined originally). I tried to define TC_p_value as global, so that each run can assign values to specific part of the array. However, it seems like I am doing something wrong here (as simulating a single array on multiple processors is not possible) or is there something wrong with my coding logic?

Any help is greatly appreciated. Code for the same is written below.

import pingouin as pg # A package to do regression
TC_p_value = np.zeros((Treecover.shape[1],Treecover.shape[2])) #let this array be of size 1000 x 100

def TC_stats(grid_start):
    global TC_p_value
    for lat in tqdm(range(grid_start, grid_start+100)):
        for lon in range(Treecover.shape[2]):
            TC_p_value[lat,lon] = pg.corr(y=Treecover[:, lat,lon].values,
                                  x=np.arange(1,16,1))['p-val'].values[0]

#Multiprocessing starts here
from multiprocessing import Pool
if __name__ == '__main__':
    pool = Pool()
    grid = np.arange(0,1000,100) #Running it in a group of 100, 10 times
    pool.map(TC_stats, grid)
    pool.close()
    pool.join()

What exactly do you mean by 'empty array'? Something about the shape? Or just that the values of `TC_p_value` remain unchanged (0)? 'empty' is a vague term when dealing in `numpy` arrays. — hpaulj, Sep 04 '20 at 15:46
@hpaulj Thanks for pointing it out. Empty array would be a wrong terminology. What you guessed is correct, it remains an unchanged 'zero' array. I'll correct this. — Ep1c1aN, Sep 04 '20 at 20:25

score 0 · Answer 1 · answered Sep 08 '20 at 03:05

The problem is that an array defined globally is not shared across processes. Thus, you need to use shared memory.

import ctypes
import numpy as np
import pingouin as pg # A package to do regression

N, M = Treecover.shape[1], Treecover.shape[2]
mp_arr = mp.Array(ctypes.c_double, N * M)
TC_p_value = np.frombuffer(mp_arr.get_obj())
TC_p_value = TC_p_value.reshape((N, M))
#let this array be of size 1000 x 100

def TC_stats(grid_start):
    TC_p_value = np.frombuffer(mp_arr.get_obj())
    TC_p_value = TC_p_value.reshape((N, M))
    for lat in tqdm(range(grid_start, grid_start+100)):
        for lon in range(Treecover.shape[2]):
            TC_p_value[lat,lon] = pg.corr(y=Treecover[:, lat,lon].values,
                                  x=np.arange(1,16,1))['p-val'].values[0]

def init(shared_arr_):
    global mp_arr
    mp_arr = shared_arr_

#Multiprocessing starts here
from multiprocessing import Pool
if __name__ == '__main__':
    pool = Pool(initializer=init, initargs=(mp_arr,))
    grid = np.arange(0,1000,100) #Running it in a group of 100, 10 times
    pool.map_async(TC_stats, grid)
    pool.close()
    pool.join()

I ran the code above with some modified toy example, and it worked.

Reference: Use numpy array in shared memory for multiprocessing

Multiprocessing a function's output to a single array

1 Answers1