Multiprocessing hangs the python kernel, doesn't work at all

Question

I got a new computer, AMD Ryzen 5800 (8cores), Win10, latest (ana)conda.

But it seems that multiprocessing doesn't work at all

import multiprocessing
import numpy as np

def my_func(A):
    return np.linalg.eig(A)

list_of_df = [np.random.rand(3000,3000),
              np.random.rand(3000,3000),
              np.random.rand(3000,3000),
              np.random.rand(3000,3000),
              np.random.rand(3000,3000),
              np.random.rand(3000,3000),
              np.random.rand(3000,3000),
              np.random.rand(3000,3000)]

with multiprocessing.Pool(processes=8) as pool:
    dfs = pool.map(my_func, list_of_df)
    print(dfs)

This code uses 1% of CPU, and fully hangs the Kernel in the JupyterLab. It doesn't respond. Whereas the following code works and it seems that numpy has some internal parallelizm, as it uses ~64% of CPU

import numpy as np

def my_func(A):
    return np.linalg.eig(A)

list_of_df = [np.random.rand(3000,3000),
              np.random.rand(3000,3000),
              np.random.rand(3000,3000),
              np.random.rand(3000,3000),
              np.random.rand(3000,3000),
              np.random.rand(3000,3000),
              np.random.rand(3000,3000),
              np.random.rand(3000,3000)]

for M in list_of_df:
    print(my_func(M))

How to make multiprocessing work again?

Multiprocessing hangs the python kernel, doesn't work at all

0 Answers0