1

I'm trying to embed python into a C application. For the moment, I am trying to get the following hello world style example working

#include <..../anaconda3/include/python3.7m/Python.h>  // I've abbreviated this path for privacy
int main()
{
    Py_Initialize();
    PyRun_SimpleString("from time import time,ctime\nprint('Today is', ctime(time()))\n");
    Py_FinalizeEx();
    return(EXIT_SUCCESS);
}

I've been able to compile this example but I get the following error when I run it

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

I have multiple versions of python installed (python3, I'm using macOS) and want to run a specific anaconda version that I have installed. As I understand the above problem, the reason I am getting this error is because I need to give a the specific path for python to look for libraries/modules. Setting PYTHONHOME and/or PYTHONPATH should fix this problem then. However, I'm not sure what I should set this value to.

My question is two fold. (1) Have I correctly diagnosed the problem? (2) If so, what should I set these two environment variables to? Specifically, what are the specific paths? While there have been several other posts regarding this problem none seem to give the what the paths should be (I've tried setting just the PYTHONHOME variable to "..../anaconda" since one of the answers in this post stated that it should be set to the parent folder of the bin file of python which in this case is anaconda. Doing this did not fix the problem.).

Edit:

Applying the changes suggested by @John Bollinger seems to partially solve the problem. The resulting error is now

Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'
HXSP1947
  • 1,311
  • 1
  • 16
  • 39

2 Answers2

2

As I understand the above problem, the reason I am getting this error is because I need to give a the specific path for python to look for libraries/modules. Setting PYTHONHOME and/or PYTHONPATH should fix this problem then.

[...]

Have I correctly diagnosed the problem?

It sounds like it. The Python interpreter chooses the default module path based either on PYTHONHOME or on its installation location and compile-time configuration, but the latter is out the window when you embed the interpreter in another program. The interpreter uses PYTHONPATH to identify additional directories to search for modules.

You should not normally set PYTHONHOME when using the standalone interpreter, but it is reasonable to do so when embedding the interpreter.

If so, what should I set these two environment variables to?

The output of python3 -h on my system includes this:

PYTHONPATH   : ':'-separated list of directories prefixed to the
               default module search path.  The result is sys.path.
PYTHONHOME   : alternate <prefix> directory (or <prefix>:<exec_prefix>).
               The default module search path uses <prefix>/pythonX.X.

Note in particular the hint about what Python expects to find in a directory named by PYTHONHOME.

In your case, you should probably set PYTHONHOME because the interpreter does not recognize how to find the system modules. The actual value that variable should take depends on where the desired Python implementation resides (and this should probably be the same implementation whose python library you linked your program against). For the system Python, on my machine, it would be

PYTHONHOME=/usr/lib:/usr/lib64

For my Anaconda 2, installed in /opt/anaconda2, it would be

PYTHONHOME=/opt/anaconda2/lib

You should not need to set PYTHONPATH (and, in fact, you might want to ensure it is unset) unless there are additional locations you want Python to search for modules.

Community
  • 1
  • 1
John Bollinger
  • 160,171
  • 8
  • 81
  • 157
  • @ John Bollinger, thanks for the response. I changed PYTHONHOME to be "..../anaconda/lib" and this seems to partially solve the problem. I'm still getting an error though (see my edit). – HXSP1947 Mar 22 '19 at 19:32
  • @HXSP1947, most likely your PYTHONHOME is still wrong, but the interpreter omits some of the diagnostic messages because it's set at all. I couldn't validate your setting even if you gave it in full, because the details are system-specific, but (1) do be certain to give an *absolute* path; (2) make sure you don't need to use the `:` form; (3) your safest bet is to point `PYTHONHOME` to the Python implementation whose libraries you linked; and (4) be sure the dynamic linker is selecting the correct libpython3.X (the `ldd` program can help with that). – John Bollinger Mar 22 '19 at 19:59
  • @ John_Bollinger, thanks for your help. I was able to evidentially find a solution (albeit not a very good one). I posted it below if you're curious. However, I accepted your answer as correct since it is the "correct way". – HXSP1947 Mar 22 '19 at 23:41
1

First, credit for pointing me down the path to a (admittedly very, very hacky) solution to this problem goes to @John_Bollinger. Second, I'm posting this answer only to serve as a solution to those that are unable to solve the problem the correct way (see @John_Bollinger answer).

I had multiple versions of python on my machine, all installed locally on my profile. When I defined "PYTHONHOME" I did so in my ".bash_profile". I was using a IDE to develop a C application with embedded python code. When I ran the application, I would get the error that I got above. The first problem was that the function "Py_Initialize()" used to start the python interpreter uses the path stored in PYTHONHOME to find the location of the interpreter to start. Since I had defined PYTHONHOME only locally via my .bash_profile what happened was that "Py_Initialize()" didn't initialize correctly.

The second problem was related to the interpreter finding modules. After correcting the above problem the interpreter was not able to find the basic core modules needed for it to run. I still don't know why (I suspect it's related to the first problem though). My solution was to manually specify the paths to the modules that I wanted to use. To get these, run the python interpreter and record the output of the following code:

import sys
print(sys.path)

Put every path that is output from here as part of the path in the C code. The full working solution is then

#include <..../anaconda3/include/python3.7m/Python.h>    

int main(int argc, char **argv, char **envp)
{

   putenv("PYTHONHOME=<path to python interpreter>");  // location of interpreter.  In my case ..../anaconda3/bin/python3.7m
   Py_SetPath(L"<paths to python modules>");  // output from print(sys.path) above in normal path format; i.e. path1:path2:...
   Py_Initialize();
   // do your stuff
   Py_FinalizeEx();  // close the interpreter and free the memory its using
}
HXSP1947
  • 1,311
  • 1
  • 16
  • 39