Error while targeting a Julia function into multiprocessing.Process of Python

Question

I am trying to parallelize a code in python by using multiprocessing.Process which targets a Julia function.

The function works fine when I call it directly, i.e. when I execute:


if __name__ == "__main__":
    import julia
    julia.Julia(compiled_modules=False)
    julia.Pkg_jl.func_jl(*args)

However, I have an error when I define the same function as a target in a Process function.

This is the code:

from multiprocessing import Process
import julia
julia.Julia(compiled_modules=False)

class JuliaProcess(object):
...

    def _wrapper(self, *args):
        ret = julia.Pkg_jl.func_jl(args)
        self.queue.put(ret) # this is for save the result of the function
    
    def run(self, *args):
        p = Process(target=self._wrapper, args=args)
        self.processes.append(p) # this is for save the process job
        p.start()
...


if __name__ == "__main__":
    ...
    Jlproc = JuliaProcess()
    Jlproc.run(some_args)

The error is when the Process starts, with the following output:

fatal: error thrown and no exception handler available.
ReadOnlyMemoryError()
unknown function (ip: 0x7f9df81cb8f0)
...

If I try to compile the julia modules in the _wrapper function, i.e.:


from multiprocessing import Process
import julia

class JuliaProcess(object):
...

    def _wrapper(self, *args):
        julia.Julia(compiled_modules=False)
        ret = julia.Pkg_jl.func_jl(args)
        self.queue.put(ret) # this is for save the result of the function
    
    def run(self, *args):
        p = Process(target=self._wrapper, args=args)
        self.processes.append(p) # this is for save the process job
        p.start()
...

if __name__ == "__main__":
    ...
    Jlproc = JuliaProcess()
    Jlproc.run(some_args)

I have the following error:

raise JuliaError(u'Exception \'{}\' occurred while calling julia code:\n{}'
julia.core.JuliaError: Exception 'ReadOnlyMemoryError' occurred while calling julia code:
const PyCall = Base.require(Base.PkgId(Base.UUID("438e738f-606a-5dbb-bf0a-cddfbfd45ab0"), "PyCall"))
...

Does anyone know what is happening? and whether it is possible using python to parallelize julia functions as I suggest.

score 1 · Answer 1 · answered Nov 18 '22 at 12:58

I finally solved the error.

The syntaxis is not the problem, but the instance on which Julia packages are precompiled.

In the first code, the error is in the call [Jl]:

julia.Julia(compiled_modules=False)

just before Julia is imported.

The second code works fine since the expression [Jl] is precompiled in the target process.

Below, I share an example that works fine if you have Julia and PyCall duly installed.

#!/usr/bin/env python3
# coding=utf-8

from multiprocessing import Process, Queue
import julia

class JuliaProcess(object):
    def __init__(self):
        self.processes = []
        self.queue = Queue()

    def _wrapper(self, *args):
        julia.Julia(compiled_modules=False)
        from julia import LinearAlgebra as LA
        ret = LA.dot(args[0],args[1])
        self.queue.put(ret) # this is for save the result of the function

    def run(self, *args):
        p = Process(target=self._wrapper, args=args)
        self.processes.append(p) # this is for save the process job
        p.start()

    def wait(self):
        self.rets = []
    
        for p in self.processes:
            ret = self.queue.get()
            self.rets.append(ret)

        for p in self.processes:
            p.join()

if __name__ == "__main__":
    jp = JuliaProcess()
    jp.run([1,5,6],[1,3,2])
    jp.wait()
    print(jp.rets)

Error while targeting a Julia function into multiprocessing.Process of Python

1 Answers1