Fatal Python error when using a dynamic version of Python to execute embedded python code

Question

SPOILER: partially solved (see at the end).

Here is an example of code using Python embedded:

#include <Python.h>
int main(int argc, char** argv)
{
    Py_SetPythonHome(argv[1]);
    Py_Initialize();
    PyRun_SimpleString("print \"Hello !\"");
    Py_Finalize();
    return 0;
}

I work under Linux openSUSE 42.2 with gcc 4.8.5 (but I also have the same problem on openSUSE 13.2 with gcc 4.8.3 or RedHat 6.4 with gcc 4.4.7).

I compiled a static and a dynamic version of Python 2.7.9 (but I also have the same problem with Python 2.7.13).

I compile my example linking to the static version of Python with the following command:

g++ hello.cpp -o hello \
-I /home/caduchon/softs/python/2.7.9/64/gcc/4.8.5/static/include/python2.7 \
-L /home/caduchon/softs/python/2.7.9/64/gcc/4.8.5/static/lib \
-l python2.7 -l pthread -l util -l dl

If I execute my example with the static version of Python in argument, it works.

If I execute it on the dynamic version of Python in argument, I get the following error (it happens in Py_Initialize()):

> ./hello /home/caduchon/softs/python/2.7.9/64/gcc/4.8.5/dynamic
Fatal Python error: PyThreadState_Get: no current thread
Aborted (core dumped)

I have no idea why it works with static version and it doesn't with the dynamic one. How can I solve this kind of problem ?

EDIT: my script installing Python is the following:

#!/bin/bash

WORKDIR=/home/caduchon/tmp/install_python_2_7_13
ARCHIVEDIR=/home/caduchon/downloads/python
PYTHON_VERSION='2.7.13'
EZ_SETUP_VERSION='0.9'
SETUPTOOLS_VERSION='34.1.0'
CYTHON_VERSION='0.25.2'
NUMPY_VERSION='1.12.0'
SCIPY_VERSION='0.18.1'
MATPLOTLIB_VERSION='2.0.0'
INSTALLDIR=/home/caduchon/softs/python/$PYTHON_VERSION/64/gcc/4.8.5
LAPACKDIR=/home/caduchon/softs/lapack/3.6.1/64/gcc/4.8.5

### Tkinter ###
echo "Install Tkinter"
sudo apt-get install tk-dev

### Workdir ###
echo "Create workdir"
mkdir -p $WORKDIR/static
mkdir -p $WORKDIR/dynamic

### Python
for x in static dynamic
do
    echo "Install Python ($x)"
    cd $WORKDIR/$x
    echo "  extract archive"
    cp $ARCHIVEDIR/Python-$PYTHON_VERSION.tgz .
    tar -xzf ./Python-$PYTHON_VERSION.tgz &> archive.log
    cd ./Python-$PYTHON_VERSION
    echo "  configure"
    if [ "$x" = "static" ]
    then
        ./configure --prefix=$INSTALLDIR/$x --libdir=$INSTALLDIR/$x/lib &> configure.log
    else
        export LD_RUN_PATH=$INSTALLDIR/$x/lib
        ./configure --enable-shared --prefix=$INSTALLDIR/$x --exec-prefix=$INSTALLDIR/$x --libdir=$INSTALLDIR/$x/lib &> configure.log
    fi
    echo "  build"
    make &> make.log
    echo "  install"
    make install &> make_install.log
    echo "  done"
done

### setuptools
for x in static dynamic
do
    echo "Install setuptools ($x)"
    cd $WORKDIR/$x
    echo "  extract archives"
    cp $ARCHIVEDIR/ez_setup-$EZ_SETUP_VERSION.tar.gz .
    tar -xzf ./ez_setup-$EZ_SETUP_VERSION.tar.gz &> archive.log
    cp $ARCHIVEDIR/setuptools-$SETUPTOOLS_VERSION.zip .
    unzip ./setuptools-$SETUPTOOLS_VERSION.zip &> archive.log
    cp ./ez_setup-$EZ_SETUP_VERSION/ez_setup.py ./setuptools-$SETUPTOOLS_VERSION/.
    cd ./setuptools-$SETUPTOOLS_VERSION
    echo "  install"
    $INSTALLDIR/$x/bin/python ./ez_setup.py &> setup.log
    echo "  done"
done

### Cython
for x in static dynamic
do
    echo "Install Cython ($x)"
    cd $WORKDIR/$x
    echo "  extract archive"
    cp $ARCHIVEDIR/Cython-$CYTHON_VERSION.tar.gz .
    tar -xzf ./Cython-$CYTHON_VERSION.tar.gz &> archive.log
    cd ./Cython-$CYTHON_VERSION
    echo "  install"
    $INSTALLDIR/$x/bin/python ./setup.py install &> install.log
    echo "  done"
done

### NumPy
for x in static dynamic
do
    echo "Install NumPy ($x)"
    cd $WORKDIR/$x
    echo "  extract archive"
    cp $ARCHIVEDIR/numpy-$NUMPY_VERSION.zip .
    unzip ./numpy-$NUMPY_VERSION.zip &> archive.log
    cd ./numpy-$NUMPY_VERSION
    echo "  build"
    $INSTALLDIR/$x/bin/python ./setup.py build --fcompiler=gfortran &> build.log
    echo "  install"
    $INSTALLDIR/$x/bin/python ./setup.py install &> install.log
    echo "  done"
done

### SciPy
for x in static dynamic
do
    echo "Install SciPy ($x)"
    cd $WORKDIR/$x
    echo "  extract archive"
    cp $ARCHIVEDIR/scipy-$SCIPY_VERSION.tar.gz .
    tar -xzf ./scipy-$SCIPY_VERSION.tar.gz &> archive.log
    cd ./scipy-$SCIPY_VERSION
    echo "  configure"
    echo "[DEFAULT]" > ./site.cfg
    echo "library_dirs = $LAPACKDIR/lib64" >> ./site.cfg
    echo "search_static_first = true" >> ./site.cfg
    echo "  build"
    $INSTALLDIR/$x/bin/python ./setup.py build --fcompiler=gfortran &> build.log
    echo "  install"
    $INSTALLDIR/$x/bin/python ./setup.py install &> install.log
    echo "  done"
done

### MatPlotLib
for x in static dynamic
do
    echo "Install MatPlotLib ($x)"
    cd $WORKDIR/$x
    echo "  extract archive"
    cp $ARCHIVEDIR/matplotlib-$MATPLOTLIB_VERSION.tar.gz .
    tar -xzf ./matplotlib-$MATPLOTLIB_VERSION.tar.gz &> archive.log
    cd ./matplotlib-$MATPLOTLIB_VERSION
    echo "  build"
    $INSTALLDIR/$x/bin/python ./setup.py build &> build.log
    echo "  install"
    $INSTALLDIR/$x/bin/python ./setup.py install &> install.log
    echo "  done"
done

EDIT: I identified a possible cause of the problem. If I remove the line export LD_RUN_PATH=$INSTALLDIR/$x/lib in the installation of dynamic Python, my embedded code works. I printed sys.path through embedded code and it point to the right installation. BUT... in this way I can't use the installation directly : it loads a wrong version found in the system (when I print sys.path I see it points to /usr/...). Also, I don't want to have to set environment variables to launch Python because I use several versions of Python on the same machine.

EDIT: Keeping my default installation script of Python, I solve the problem by adding -rdynamic in the options when compiling the C++ example. But I don't well understand what is this option, and which kind of disaster it can cause...

Answer 1

If I understand correctly, you want to run the statically linked version while setting the Python home to the dynamically linked version. This doesn't work.

Here's what happens: when you run the Py_Initialize() of the statically linked library, it will at some point try to import the _locale module. Because you set the Python home to the dynamically linked version, it will load $INSTALLDIR/dynamic/lib/python2.7/lib-dynload/_locale.so. This library is dynamically linked against $INSTALLDIR/dynamic/lib/libpython2.7.so.1.0. Now you end up with two copies of the interpreter. The first copy is the statically linked one, which is being initialized. The second copy is uninitialized. When the dynamic module importing mechanism tries to initalize the _locale module, it fails because _locale's init function is referring to the second, completely uninitialized interpreter.

What was the reason you tried this? If you tell us which problem you wanted to solve in the first place, we might be able to help you.

EDIT: (I wrote this after the first edit, I didn't try so far what happens with -rdynamic): When you don't set LD_RUN_PATH, $INSTALLDIR/dynamic/lib/python2.7/lib-dynload/_locale.so is dynamically linked against the system's libpython2.7.so. The reason you don't see an error is that importing the _locale module fails with an ImportError (instead of segfaulting), but this ImportError is catched during interpreter initialization (while previously the segfault couldn't be catched). But if you try in the embedded interpreter to import _locale (or any other extension module like for example _struct), you get an error like this:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: $INSTALLDIR/dynamic/lib/python2.7/lib-dynload/_locale.so: undefined symbol: PyUnicodeUCS2_FromObject

EDIT: When compiling hello.cpp against the static Python version, normally most symbols like _PyThreadState_Current don't end up in the dynamic symbol table. This is why you end up with "two copies of the interpreter" as described above and the segfault. However, when passing -rdynamic, these symbols end up in the dynamic symbol table, so now the module init function from the _locale.so of the "dynamic" build refers to _PyThreadState_Current of the "static" build. I'm still not convinced that what you try to do (using a program linked against the "static" build with the Python home of the "dynamic" build) is a good idea, though. ;)