2

There is this stackoverflow post that really nicely shows a way to calculate the proximity matrix of a RandomForestClassifier().

Proximity Matrix in sklearn.ensemble.RandomForestClassifier

Nevertheless the for-loop in that script is quite slow if you have a large dataframe. I tried to parallelize this for-loop, but unsuccesfully. I only get 'None' as an output.

How can I parallelize this for-loop in Spyder 4 running Python 3.8.5 on Windows 10?

proxMat = 1*np.equal.outer(a, a)

for i in range(1, nTrees):
      a = terminals[:,i]
      proxMat += 1*np.equal.outer(a, a)
JonnDough
  • 827
  • 6
  • 25

1 Answers1

1

Here you want to perform a reduce operation - so parrallelization is not obvious. You did not specify how you tried to parallelize the loop. A simple way to parrallelize :

import multiprocessing
pool = multiprocessing.Pool(processes=4)

def get_outer(i):
   return np.equal.outer(terminals[:,i],terminals[:,i])

todo = list(range(1, nTrees))
results = pool.map(get_outer, todo)
proxMat = 1*np.equal.outer(a, a)
for res in results:
    proxMat+ = res

I'm not sure this one would help, but possibly you'd have less pickling problems :

import multiprocessing
pool = multiprocessing.Pool(processes=4)

def get_outer(t):
   return np.equal.outer(t,t)

# This part might be costly !
terms = [terminals[:,i] for i in range(1, nTrees)]

results = pool.map(get_outer, terms)
proxMat = 1*np.equal.outer(a, a)
for res in results:
    proxMat+ = res
SergeD
  • 44
  • 9
  • Hi SergeD, Thank you for your suggestion, however the results throws an error: 'Can't pickle : attribute lookup on__main__ failed' – JonnDough Mar 17 '21 at 10:57
  • 1
    Try declaring the function by another mean (regular function declaration, etc). If you have a well defined function, multiprocessing.pool.map should work. I edited the code. – SergeD Mar 18 '21 at 12:34
  • There are many discussions it seems on how to pass several arguments to pool.map. Maybe check here :https://stackoverflow.com/questions/5442910/how-to-use-multiprocessing-pool-map-with-multiple-arguments – SergeD Mar 18 '21 at 12:38