Fatal error in extension: PyThreadState_Get: no current thread

Question

I've seen several posts that have stated the same error, but looking and trying out the answers in those posts have not helped. I was wondering if someone could look at this and see if something pops out?

I'm building a Python extension for a CPP application, and there are no errors during the compilation and build step. However, when I import the module I get the error mentioned in the title. Other stackoverflow answers have claimed that this is because of being linked with one library while compilation and using a different interpreter. As far as I can tell, I'm using the same Python interpreter. I'm going to describe now why I think I'm using the same Python in the linking process and for the interpreter.

This is the comand I'm using to build the Python extension

$ gcc -shared helicsPYTHON_wrap.c $(python-config3 --includes) -I/path/to/helics-0.9/includes -L/path/to/helics-0.9/lib -lhelicsSharedLib -L$(python3-config --prefix)/lib -lpython3.6m -o _helics.so

$ which python3-config
/Users/$USER/miniconda3/bin/python3-config

$ python3-config --prefix
/Users/$USER/miniconda3

If I try to import the python file that imports the shared library, it throws the fatal error. If I use otool -L on the shared library, I get the following. This is what I expect to get.

$ otool -L _helics.so
_helics.so:
        @rpath/libhelicsSharedLib.dylib (compatibility version 0.0.0, current version 0.0.0)
        @rpath/libpython3.6m.dylib (compatibility version 3.6.0, current version 3.6.0)
        /usr/local/opt/zeromq/lib/libzmq.5.dylib (compatibility version 7.0.0, current version 7.3.0)
        libboost_program_options.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_filesystem.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_date_time.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/local/opt/gcc/lib/gcc/7/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.24.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
        /usr/local/lib/gcc/7/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

I also tried install_name_tool to add the full path of the libpython3.6m.dylib.

$ install_name_tool -change @rpath/libpython3.6m.dylib /Users/$USER/miniconda3/envs/py3/lib/libpython3.6m.dylib _helics.so

I still get the same fatal error. My hypothesis is that my Mac System Python 2.7 installation is having an effect on this process at some stage. I'm unable to identify where though.

Is there a way to add more debug statements to find out why there is a Fatal Python error. Currently, the error message is very short.

$ python helics.py
Fatal Python error: PyThreadState_Get: no current thread

[1]    64481 abort      python helics.py

Curiously, if I use a conda environment and use Python 2.7, I'm able to load the extension fine! This is why I think that when I'm using Python 3.6, it is somehow picking up something from the default mac system python 2.7 installation and working fine. It is picking the same thing up when I use the conda 2.7 python environment, but because they are both Python 2.7 (though conda is 2.7.14 and system python is 2.7.10) it seems to work. This is the otool -L output when I use a conda environment.

$ otool -L _helics.so
_helics.so:
        @rpath/libhelicsSharedLib.dylib (compatibility version 0.0.0, current version 0.0.0)
        @rpath/libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
        /usr/local/opt/zeromq/lib/libzmq.5.dylib (compatibility version 7.0.0, current version 7.3.0)
        libboost_program_options.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_filesystem.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_date_time.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/local/opt/gcc/lib/gcc/7/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.24.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
        /usr/local/lib/gcc/7/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

The questions I have are 1) how do I get more debug information out of the error from Python. I have tried python -vvv and that does not give me enough information. I tried using gdb but that also did not give me any information. I believe it requires recompiling Python itself using debug symbols. 2) Do you have any recommendations on how to solve this problem or debug further?

Also, I'm not sure if this is useful information, but I am able to use ctypes and load the shared library after I create it. I'm just unable to import it as a python module.

This is the original issue if one is interested - https://github.com/GMLC-TDC/HELICS-src/issues/59

Edit: I tried this using zsh and bash, and still got the same error. I also tried setting the following export PATH="/Users/$USER/miniconda3/bin:/Users/$USER/miniconda3/lib" temporarily in the shell and running and I STILL get the same error. This should have excluded my Mac System Python 2.7.10, so I'm really not sure what is going on.

Edit again: I've also tried reinstalling miniconda with Python2. And if I use Python2 everything works fine. I'm just unable to use Python3 using miniconda. Oddly enough, if I use homebrew and install Python3 that seems to work fine.

Edit again: ~~This is possibly an issue with High Sierra. I currently don't have access to another mac, but I'm on the latest operating system which has SIP. I'm not sure if this is causing this issue.~~ Additionally, I've tried using Anaconda3 and had no luck.

Edit again: This does not seem to be related to the operating system. I'm able to run this successfully on another computer with High Sierra.

Edit again: I tested this on other fresh OS installs, and they don't work. But they do work on two of my machines. Are there other tools that tell you what dependency a library requires or where Python throws a fatal error? My best guess at the moment is that I've installed something on my other machines in the past that allows this to work. I need to identify what that was and make sure I can document it.

Edit again: I've added a gist of the output of the version of Python that I'm using.

Edit again: I've added the tags for miniconda and anaconda since I don't experience this issue when using homebrew python3, but only just when I'm using miniconda3 or anaconda2 with a python3 environment. This always appears to work with Python2, regardless of whether it is homebrew, anaconda or miniconda.

Edit again:

These are the steps if someone else wants to replicate on their machine.

git clone https://github.com/GMLC-TDC/HELICS-src
mkdir build-osx
brew install boost
brew install cmake
brew install swig
cmake -DBUILD_PYTHON=ON -DPYTHON_LIBRARY=$(python3-config --prefix)/lib/libpython3.6m.dylib -DPYTHON_INCLUDE_DIR=$(python3-config --prefix)/include/python3.6m/ ..
make
cd ./swig/python/
python helics.py # Error

I've weirdly had this same issue when compiling threaded Python extensions using MSYS2 on Windows. Not sure if it's related, but I would like to know the answer regardless. — Alex Huszagh, Jan 08 '18 at 07:00
I've added a gist that shows the output of the commands you describe and a few others. — kdheepak, Jan 08 '18 at 20:54
No, I don't get it in Ubuntu. I tried the exact same steps and it works fine. — kdheepak, Jan 12 '18 at 03:33
Add in to `helics.py`(end of script), `import time; while True: time.sleep(1);` — dsgdfg, Jan 12 '18 at 07:44
I still get the same error because `import _helics` is the line it fails at. — kdheepak, Jan 13 '18 at 08:56

score 4 · Accepted Answer · answered Jan 27 '18 at 10:12

I was able to solve this problem by changing the CMakeLists.txt to use -undefined dynamic_lookup as suggested in this answer. E.g. of the CMakeLists.txt is here. And the reason I was getting different outcomes on different machines was because one of my macs had Python 3.6.1 but the others had Python>=3.6.2

score 3 · Answer 2 · answered Jan 13 '18 at 17:10

3

You have Python 3.6, from homebrew... While your module, while building referenced Python 2.7, provided by the system. Here is same problem described. From one of the comments - python3.6-config --ldflags will show LDFLAGS to be used in your Makefile.

answered Jan 13 '18 at 17:10

Michał Zaborowski

3,911
2
19
39

Firstly I don't have python3.6 from homebrew, and I don't even have python2.7 from homebrew. I uninstalled both of them and I still get this error with miniconda3. I think it has to do with system python. I tried the same thing you mentioned but that doesn't seem to help. Everything I've tried seems to indicate what I'm currently doing is fine. I have the same setup on multiple machines, and it works on two of them but not on the others. – kdheepak Jan 14 '18 at 17:24
Problem happens if you have python 3.X installed - for instance by homebrew, but miniconda "works" as well, or I should write - fails... Problem is that your compiler sees headers for old python, so that has to be fixed. – Michał Zaborowski Jan 14 '18 at 19:28
How do I make it not see the System Python 2.7 headers? Is there a way to do that? Where does it look (e.g. `$PATH`, `$PYTHONPATH`) to "see" these headers? – kdheepak Jan 15 '18 at 21:54
before calling `gcc` do `export LDFLAGS=$(python3.6-config --ldflags)` – Michał Zaborowski Jan 15 '18 at 22:13
I still get the same error even if I used your LDFLAGS export tip. – kdheepak Jan 16 '18 at 20:40
[Their steps](https://github.com/GMLC-TDC/HELICS-src/blob/master/installing-on-osx.md) are a bit different, so it can be a problem. It can be that there are two `cmake` - system in `/usr/...`, and homebrew one from `/Library/Developer/CommandLineTools`. Next - pleas look [here](https://stackoverflow.com/questions/16246352/how-do-i-specify-ldflags-and-cppflags-for-configure) - but I can't see `configuration` at the repository. Looks like `LDFLAGS` are ignored, or overwritten... but why, and how - that is the question. – Michał Zaborowski Jan 16 '18 at 21:50
I am part of the team that is building the Python extension for HELICS. I was involved in putting together that example repo as well. I'm fairly sure that what you have described isn't the problem. In fact, I was able to create a minimal working example here - https://stackoverflow.com/questions/48289858/fatal-error-in-extension-pythreadstate-get-no-current-thread-when-using-swig-w so others can test to see if they have this error, even without installing helics. – kdheepak Jan 16 '18 at 22:29
Noticed your commits, so it was clear that you are involved. Also you mentioned that it works on other systems. So problem is with linking. `cmake`/`make` fails. It takes wrong refs, but it is really hard to tell why. – Michał Zaborowski Jan 16 '18 at 22:51
If it is possible, are you able to run this mwe? https://github.com/kdheepak/mwe-swig-python3-anaconda – kdheepak Jan 17 '18 at 00:08
I don't have mac, so... can't. However I've found [blogpost on that](https://www.cprogramming.com/tutorial/shared-libraries-linux-gcc.html). Then realized, that you are doing it OK. So tracked for LD_LIBRARY_PATH. Looks like sometimes that is blocked by the system, but also found [this post](https://github.com/JuliaPy/PyCall.jl/issues/245#issuecomment-281131624). Problem is that they can rebuild python, and not sure if conda allows it as well... – Michał Zaborowski Jan 17 '18 at 08:51
Thanks for those links. It's just really odd again since I'm unable to get the error on one of my macs, where LD_LIBRARY_PATH is not set. I don't know what on that machine is different that is allows for this to work. – kdheepak Jan 17 '18 at 14:26

score 0 · Answer 3 · answered Jan 15 '18 at 06:16

0

Make sure the active python framework's lib directory is in the linker search path.Hope this would work..

answered Jan 15 '18 at 06:16

Abhi4967

1
3

Fatal error in extension: PyThreadState_Get: no current thread

3 Answers3