Julia -- jupyter kernel dying in multithread when calling numpy

Question

I have been having a problem with multithreading in Julia that I have managed to break down to the simple example:

The script

using PyCall, PyPlot
@pyimport numpy as np
N = 1000
a = np.zeros(N)
Threads.@threads for i in 1:length(a)
    #println("hello from thread ",Threads.threadid())
    a[i] = sin(i*pi/N)
end
plt.plot(a)

runs, whereas

using PyCall, PyPlot
@pyimport numpy as np
N = 1000
a = np.zeros(N)
Threads.@threads for i in 1:length(a)
    #println("hello from thread ",Threads.threadid())
    a[i] = np.sin(i*pi/N)
end
plt.plot(a)

does not. Instead the jupyter kernel dies. So apparently there is a problem using functions from pyimport inside of a parallel loop in Julia? The loop also runs well with numpy call in serial.

I am using jupyter lab on a redhat cluster. Using 32 threads. Running Julia 1.7.1

I think you should highlight that by "kernel" you refer to Jupyter, not some kernel function. — phipsgabler, Feb 04 '22 at 12:15
Python does not have multi-threads (openmp style) because GLI https://stackoverflow.com/questions/1294382/what-is-the-global-interpreter-lock-gil-in-cpython — jling, Feb 04 '22 at 13:19
Seems to be a known bug: https://github.com/JuliaPy/PyCall.jl/issues/885 — phipsgabler, Feb 04 '22 at 13:35

Oyvach · Answer 1 · 2022-02-05T08:48:17.133

0

I managed to get the example to run in parallel by using Julia's distributed package instead of the threading.

Then

Using Distributed
N = 1000
a = np.zeros(N)
function sinmake(i)
    a[i] = np.sin(i*pi/N)
end
pmap(sinmake,1:N)
plt.plot(a)

It would be nice to be able to use python packages in thread-parallelism as well, but this solves my problem for now.

edited Feb 05 '22 at 08:48

answered Feb 05 '22 at 08:09

Oyvach

11
2

But... that code doesn't do anything treaded or distributed, at all. You just imported the package. Of course it works. Did you mean to use `pmap`? – phipsgabler Feb 05 '22 at 08:31
You are right.. sorry I mean pmap – Oyvach Feb 05 '22 at 08:48

Julia -- jupyter kernel dying in multithread when calling numpy

1 Answers1