4

I have a backend algorithm which makes julia calls, I do this via the PyCall package. however, one of the packages in julia recently updated, and it itself makes use of PyCall. no mater what I try to do, I always get an error. If i set pycall in julia via Pkg.build("PyCall"), I can call the package, however python cannot call julia. if I set pycall in python3 via julia.install(), then the package can't run properly in julia.

to reproduce this here is a quick example:

test.jl

using Pkg
Pkg.add("PyCall")
ENV["PYTHON"]=""
Pkg.add(PackageSpec(url="https://bitbucket.org/SSamanipour/safd.jl/src/master/"))
Pkg.build("SAFD")
Pkg.test("SAFD")

test.py

from julia import Main as jl
jl.include("test.jl")

if I run python3 test.py

I get into an error as that specific python is not configured py the shared pycall.

what I would like is that the pycall in python not be the same for python. it seems that if you configure pycall in one it will force the other.

I tried using juliacall, however if I use pycall with it, it seems to also break.

if I try to have ENV["PYTHON"]="python3", I can't run the SAFD package.

Any help would be greatly appreciated, as I've been a few days on this and tried multiple solutions and everything seems to give me a different error.

some info: doing this on an aws amazon2 linux OS and an aws r5.large instance.

I'm using python 3.10.0

versioninfo() Julia Version 1.6.3 Commit ae8452a9e0 (2021-09-23 17:34 UTC) Platform Info: OS: Linux (x86_64-pc-linux-gnu) CPU: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-11.0.1 (ORCJIT, skylake-avx512)

EDIT:

I ended up figuring out the issue, The package I was trying to use updated its dependencies, when it builds, it automatically updates the conda python with all the packages it needs. However, if it is not built by conda, it will assume the dependencies are there. What I'm very surprised by is that the error message never indicated anything about a specific package. I only figured it out a bit randomly.

One thing is clear is that you can't have two different pythons, pycall only configures one python for back and forths between python and julia.

Flying Turtle
  • 366
  • 7
  • 20

1 Answers1

4

You seem to have 2 Python's on your system:

  • first one installed after ENV["PYTHON"]="";Pkg.build("PyCall") --> this is the default way to install PyCall in Julia to get everything working (this is actually a Miniconda Python)
  • system Python that you start with python3

Basically, mixing those 2 Pythons and having everything to work is like finding a needle in a haystack being blindfolded. Rather than that you need to use one Python for everything and to get things running smoothly that should be the Miniconda Python that got installed by Julia when doing ENV["PYTHON"]="";Pkg.build("PyCall").

In order to find Python you want to use do in your Julia console:

julia> using PyCall

julia> PyCall.python
"/home/ubuntu/.julia/conda3/python.exe"

Now you found the Python code that you need! So what you run in bash is:

/home/ubuntu/.julia/conda3/python.exe test.py
Przemyslaw Szufel
  • 40,002
  • 3
  • 32
  • 62
  • 1
    Can you provide a little information on why 2 different Pythons can't coexist in this case? I can imagine two active Python interpreters not being able to share data, but the code doesn't seem to try to make them communicate. Is the `ENV["PYTHON"]` being set to the Miniconda Python somehow throwing off the active CPython? – BatWannaBe Nov 11 '21 at 17:03
  • 2
    They can but basically all libraries need to match. This is just my empirical experience - if there are slightest differences in libraries and you do cross-python everything stops working. Here you have a worst situation of all - one Python is Miniconda and the other Python is system a Python. I am not a Python expert but whenever I tried kind of mixing it always ends with Python library hell. For an example Python Anaconda has an inbuild `pip`. You can do on Anacpnda `pip install numpy` - if you do that everything in Python Anaconda stops working (another personal empirical observation) – Przemyslaw Szufel Nov 11 '21 at 17:37