Is it possible to modify PYTHONPATH at runtime?

Question

I have a C++ application dynamically linked to the Python interpreter. I want to be able to import python modules from a particular directory. I want to modify the PYTHONPATH for my process so that sys.path will include the paths that I added to the PYTHONPATH. That seems to be the way it works according to this documentation:

http://docs.python.org/c-api/intro.html#embedding-python

However, when I print sys.path from Python-land it has the original contents of PYTHONPATH and not the one I set. Here's an example of what I'm doing (using Boost.Python):

int main(int argc, char* argv[])
{
  _putenv_s("PYTHONPATH", "C:\\source\\\\modules");
  Py_Initialize();
  object main = import("__main__");
  object global = (main.attr("__dict__"));
  exec("import sys\nprint sys.path"), global, global);
}

PS - I know there are other ways to accomplish my goal, but that's not what I'm asking about. I am wondering why Py_Initialize() doesn't use the current value of PYTHONPATH when setting up sys.path. Or perhaps I've misunderstood how it is supposed to work?

Answer 1

I found cross-platform solution. Before invoke any other python code just execute following python lines:

import sys
sys.path.append("C:\\source\\\\modules")

Answer 2

Check out:

void PySys_SetPath(char *path) Set sys.path to a list object of paths found in path which should be a list of paths separated with the platform’s search path delimiter (: on Unix, ; on Windows).

or

Py_SetProgramName(argv[0]); That adds dirname(argv[0]) to your PYTHONPATH for you.

Answer 3

This happens if you're using more than one C runtime library at a time. In this case, your application and the Python DLL are probably linked against different CRTs. Each CRT has its own set of environment variables; changes to the environment made with putenv from one CRT are not visible from getenv calls made with a different CRT.

See the "readEnv" example at http://msdn.microsoft.com/en-us/library/ms235460%28v=vs.80%29.aspx.

You can fix this by making sure to use only a single CRT, but that's tricky in practice. Debug builds of programs typically use debug CRTs (which enable things like heap checks and API assertions); production DLLs, even when used in debugging, typically use MSVCRT, the production, threadsafe version. I've worked around this by disabling the debug CRTs entirely, setting all builds to "multithreaded dynamic", since maintaining separate debug DLLs is too much of a hassle. You lose some debugging capabilities by doing that.

Answer 4

You could just do python3 -c "import sys; sys.path.append('C:\Path\To\Modules')"

Answer 5

As other people have said, you may be running into a CRT mismatch. I was able to get this to work with Python 2.6 and Visual C++ 2008:

#include "stdafx.h"
#include "Python.h"

int _tmain(int argc, _TCHAR* argv[])
{
  _putenv_s("PYTHONPATH", "C:\\source\\\\modules");
  Py_Initialize();
  PyRun_SimpleString("import sys\nprint sys.path");
  PyRun_SimpleString("raw_input()");
  return 0;
}

This output:

['C:\\Python26\\lib\\site-packages\\distribute-0.6.13-py2.6.egg', 'C:\\Python26\
\lib\\site-packages\\virtualenv-1.4.9-py2.6.egg', 'C:\\source\\modules', ...

Another option may be to change to that directory, since the current directory typically ends up on the path, e.g.:

    _chdir("c:\\");
    Py_Initialize();
[...]

which gives me:

['C:\\Python26\\lib\\site-packages\\distribute-0.6.13-py2.6.egg', 'C:\\Python26\
\lib\\site-packages\\virtualenv-1.4.9-py2.6.egg', 'C:\\Windows\\system32\\python
26.zip', 'C:\\Python26\\Lib', 'C:\\Python26\\DLLs', 'C:\\Python26\\Lib\\lib-tk',
 'c:\\', ...

Answer 6

#include "Python.h"
int main()
{
  Py_Initialize();
  PyRun_SimpleString("import sys");
  PyRun_SimpleString("sys.path.append(\"<some_path>\")");
  return 0;
}

This worked for all python version (2.6, 2.7, 3.1, 3.2, 3.3 and 3.4).
Several notes regarding <some_path>:

• It should contain only a single directory. List of directories with valid separators (d:/path1;d:/path2 etc.) is not working

• Windows paths like: d:\\path1 will work only for python versions prior to Python 3, for later versions d:\\\\path1 should be used. I'd advice replacing windows path delimiters with unix delimiters. Following code chunk does this.

  std::string my_path = "<some_path>"; std::replace(my_path.begin(), my_path.end(), '\\', '/');

A gentle advise: Don't waste your time trying to modify PYTHONPATH with either of the following API methods if you want to support different python versions:

• Py_SetPythonHome() - for python 2 requires an ascii string, for python 3 - a unicode string, but doesn't work reliably for versions greater than 3.1
• Py_SetPath() - introduced in python 3, but is buggy (see http://bugs.python.org/issue11320)

In general API methods listed above do not affect after Py_Initialize() call.

Answer 7

It is possible that the Python DLL gets its own copy of the environment when it is loaded. Try loading it with LoadLibrary and GetProcAddress after you've changed the environment and see if that changes anything.