2

I am using subprocess.Popen to call an external Python script from my installed Application, bundled using PyInstaller. The format of this command is something like this:

subprocess.Popen(["/usr/bin/python", "/path/to/exe/SDK.py"],
                 stdin=subprocess.PIPE, stdout=subprocess.PIPE,
                 stderr=subprocess.PIPE, shell=False)

I have successfully tested this on both Windows and MacOS - both can run the external script. However, on other Posix OS' I get the following error:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
ImportError: No module named site

Now, I realize that this question has been asked before, and generally it is down to PYTHONHOME having an incorrect value. However, if I run the command in a format similar to:

PYTHONHOME=/usr /usr/bin/python /path/to/exe/SDK.py

I get no errors in the logs, but the script SDK.py doesn't execute.

Therefore to prove that this isn't an issue with my script; I installed my own version of Python onto the machine - after doing this, the script executed successfully. I tested with both /usr/bin/python and /home/vagrant/Python-2.7.15/python, without needing to specify a PYTHONHOME in the subprocess command.

However, I still need to allow users to execute the SDK.py script using the built in, OS version of Python.

So to do this I have tried these other things:

  1. Using sys.executable in the subprocess.Popen call to execute SDK.py.

(The value of which was: /opt/program_name/lib/program_name)

  1. Forcing PYTHONPATH & PYTHONHOME to be empty by explicitly setting the environment:

    import os
env = os.environ.copy()
env['PYTHONHOME'] = ''
env['PYTHONPATH'] = ''
subprocess.Popen(["/usr/bin/python", "/path/to/exe/SDK.py"], env=env)

Can anyone explain how I can call the system version of Python from an external subprocess like this?

EDIT: Output when calling Python in verbose mode (from the CLI):

>>> from sys import executable
>>> executable
'/usr/bin/python'
>>> import _csv
# trying _csv.so
# trying _csvmodule.so
# trying _csv.py
# trying _csv.pyc
# trying /usr/lib64/python2.7/_csv.so
# trying /usr/lib64/python2.7/_csvmodule.so
# trying /usr/lib64/python2.7/_csv.py
# trying /usr/lib64/python2.7/_csv.pyc
# trying /usr/lib64/python2.7/plat-linux2/_csv.so
# trying /usr/lib64/python2.7/plat-linux2/_csvmodule.so
# trying /usr/lib64/python2.7/plat-linux2/_csv.py
# trying /usr/lib64/python2.7/plat-linux2/_csv.pyc
# trying /usr/lib64/python2.7/lib-dynload/_csv.so
dlopen("/usr/lib64/python2.7/lib-dynload/_csv.so", 2);
import _csv # dynamically loaded from /usr/lib64/python2.7/lib-dynload/_csv.so
tmccaffrey
  • 31
  • 4
  • What is the value of your PYTHONHOME initially? And the path you tried isn't correct, the path components need to be separated by a colon, not a blank. – Markus Sep 16 '18 at 16:27
  • @markus Initially, on a fresh vm, whenever I did `echo $PYTHONHOME` it gives an empty, blank line. So I think that its not set. – tmccaffrey Sep 16 '18 at 16:31
  • As for the path being incorrect, do you mean the one where I specify the `PYTHONHOME=/usr/`? – tmccaffrey Sep 16 '18 at 16:35
  • Yes, that one. But I just realized that I misread that line. But just /usr for the value of PYTHONHOME is most likely not correct. – Markus Sep 16 '18 at 16:40
  • I think I was using this SO Comment as a pointer for setting `PYTHONHOME` to to `/usr/`: https://stackoverflow.com/a/36173084 – tmccaffrey Sep 16 '18 at 17:03
  • Just a hunch: can you try explicitly unsetting PYTHONHOME, and then run the Python command from the same command line? Does it change anything? – Markus Sep 16 '18 at 18:15
  • Yeah @Markus thanks for the help, I'll try this tomorrow morning! – tmccaffrey Sep 16 '18 at 19:01

2 Answers2

1

This is not a full answer, but for the sake of completeness I will document here the two things that worked - and which approach we went with in the end.

  1. For some strange reason, when my colleague did this:

env = os.environ.copy() env['LD_LIBRARY_PATH'] = env['PATH']

It allowed us to call the subprocess when we passed this env into the Popen() call like so:

self.process = subprocess.Popen(sdk_executable_command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False, env=env)

We did not investigate this further because we went with the second solution described below. However, it would lead me to believe that these issues may have been down to the predefined setting of LD_LIBRARY_PATH.

  1. When we created a virtualenv for python using the pip package of the same name, and we called the SDK using this python executable - the SDK would successfully run. Reason for this is still unknown, but because this is an SDK: we decided it was appropriate for the eventual SDK Developer to run this in an environment separate from the 'out of the box' Python in Linux.
tmccaffrey
  • 31
  • 4
0

I'll try and walk you trough the process of checking you Python installation on CentOS 7.

First of all, open some terminal window and verify the installation of the python RPMs:

$ rpm -V python python-libs

If you get no output, everything is fine. Next check the environment variables:

$ env | grep PYTHON

There should be no output as well. If there is some mention of PYTHONHOME or PYTHONPATH, unset them. Now check what Python thinks about its default path:

$ /usr/bin/python
Python 2.7.5 (default, Jul 13 2018, 13:06:57) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from sys import path
>>> print path
['', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages']

You should also be able to import the site module without any problems:

>>> import site
>>> print site
<module 'site' from '/usr/lib64/python2.7/site.pyc'>

If everything is OK, try to run your script from the very same command line where you performed the tests.

Markus
  • 3,155
  • 2
  • 23
  • 33
  • Hi Markus, everything is as you say here except for one thing: `/usr/lib64/python2.7/site-packages/gtk-2.0` is missing from `sys.path` Would this be relevant? – tmccaffrey Sep 17 '18 at 09:26
  • @tmccaffrey Don't think so, unless you somewhere need the GTK Python packages, of course. Do you still get the same error? – Markus Sep 17 '18 at 09:31
  • @tmccaffrey Just to avoid any misunderstanding... You don't try to set any of the `PYTHON*` variables to an empty string? By unsetting, I mean `unset PYTHONHOME`. And don't fiddle with those variables in your script. – Markus Sep 17 '18 at 09:36
  • 1
    And if you create a new (copy) of the `environ` dict you pass to your process, like in your second example, unsetting the variable would be done by means of `del(env['PYTHONHOME])` instead of `env['PYTHONHOME] = ''` which sets it to empty string. – Ondrej K. Sep 17 '18 at 11:07
  • Thanks for your comments lads, at the moment I get a KeyError if I try to `print env['PYTHONHOME']`. This should mean that it is already unset right? – tmccaffrey Sep 17 '18 at 12:24
  • @tmccaffrey: Yes, very much so. You should wrap both `del()`s in `try:`/`except KeyError:`. – Ondrej K. Sep 17 '18 at 13:09
  • Okay, we've made a bit of progress on this issue this morning. We found one more workaround and one potential solution, I'll outline both here and if the potential solution is valid I'll put up a more detailed description later. The workaround: Previously, installing another python on the machine fixed the issue, but now we've discovered that adding a virtualenv and using the python in there allows the subprocess to work. No new python needed. Potential Solution: Setting `env['LD_LIBRARY_PATH'] = env['PATH']` and use this in `Popen` seems to do the trick. Must be something to do with this. – tmccaffrey Sep 18 '18 at 11:30
  • @tmccaffrey What happens when you start Python with `python -vv` and then for example `import _csv` (with the underscore). What does it print? – Markus Sep 18 '18 at 12:00
  • You can edit it into the question, makes probably more sense. But from what I can see this look like the interpreter is trying the usual places. – Markus Sep 18 '18 at 13:34
  • It's a really strange issue, it seems like when we call the external Python from our subprocess.Popen in our App it behaves differently from when you call it in the command line. I'll keep you updated on any progress we make. – tmccaffrey Sep 18 '18 at 13:41
  • @tmccaffrey You can always use `subprocess.Popen(["/usr/bin/python", "-vv"])` to verify. – Markus Sep 18 '18 at 13:44