0

I am trying to speed up a piece of code by using concurrent.futures, however, the program does not fill the new matrix as intended. Code below:

data_new = np.random.random(6000000).reshape(100,200,300)
data_processed = np.zeros((100,200,300))
subs = [sub for sub in range(0,100)]

def processing(sub):

    detrended = signal.detrend(data_new[sub])
    std = StandardScaler()
    fit = std.fit(detrended)
    scaled_data = std.transform(detrended)
    
    data_processed[sub] = scaled_data 
    
    print(f'Subject {sub} was processed...')


t1 = time.perf_counter()

if __name__ == '__main__':
    with concurrent.futures.ProcessPoolExecutor() as executor:
        executor.map(processing, subs)
        
t2 = time.perf_counter()
print(f'Finished in {t2-t1} seconds.')

Can anyone point out the mistake in my code. the new matrix# data_processed still contains only zeros however when I use for loop the code works. Thanks in advance.

Prasoon
  • 41
  • 8
  • Do you know how operating system creates processes using fork + exec mechanism? When new process created it use his own memory, processes share nothing by default (actually there is copy on write mechanism, but it's not important in this case). I think this is the issue. I'm not sure, but likely scientific libraries works fine with Python GIL, so you could try to replace ProcessPoolExecutor with ThreadPoolExecutor. But probably you'll be heeded to use Locks to prevent race conditions. – NobbyNobbs Mar 13 '21 at 09:28
  • Thank you:. Yes ThreadPoolExecutor() works. I had the impression that ThreadPoolExecutor is used for i/o operations and ProcessPoolExecutor is for cpu intensive processes. PS: I am not a computer wizard :) and do not understand the comment fully. I will research on the race conditions, fork + exec mechanisms etc. Thank you again. – Prasoon Mar 14 '21 at 11:13
  • Does this answer your question? [Python multiprocessing with shared numpy array](https://stackoverflow.com/questions/42518609/python-multiprocessing-with-shared-numpy-array) – HTF Mar 23 '21 at 20:48

0 Answers0