2

Before I start, some notes on my environment: python 2.7.14 installed through miniconda, macOS 10.13.3.

The Problem

I'm trying to write a data processing pipeline (call it analyze.py) in python which calls several different programs. One of those programs, EMAN2 uses its own python environment to run (located, say, ~/EMAN2/bin/python). From the command line, a call to eman2 might look something like this:

~/EMAN2/bin/e2buildstacks.py <args>

where e2buildstacks.py specifies its own python environment using the standard #!/Users/<username>/EMAN2/bin/python type formulation at the top of the file (note, there's a lot of these different .py files to run, I'm just using one name as an example).

In python, I'm using a subprocess.Popen construction for these calls. A very simple example would be:

import subprocess
stacks_cmd = '/Users/<username>/EMAN2/bin/e2buildstacks.py <args>'
process = subprocess.Popen(stacks_cmd, shell=True, stdout=subprocess.PIPE)
stacks_output, stacks_error = process.communicate()

Where I've replaced the actual args and username, for simplicity.

If this is all I'm doing, it works fine. I can run python analyze.py from the command line and it runs EMAN2 without problem.

However, this pipeline is getting wrapped up in a GUI (wxpython based). So I have to use pythonw to run my program. When I try to do:

pythonw analyze.py

It can't run the EMAN2 scripts correctly, I get an error:

Traceback (most recent call last):
  File "/Users/<username>/EMAN2/bin/e2buildstacks.py", line 34, in <module>
    from EMAN2 import *
ImportError: No module named EMAN2

This indicates to me that it's using my miniconda python rather than the ~/EMAN2/bin/python to run the script.

(Note: if anyone uses EMAN2, I can provide you with the full argument list and the input files. But rest assured that's not the issue, I can run the command I'm using just fine from the command line)

What I've tried

I've tried several different things to fix this problem. The simplest was specifying the python to be used in the command:

import subprocess
stacks_cmd = '/Users/<username>/EMAN2/bin/python /Users/<username>/EMAN2/bin/e2buildstacks.py <args>'
process = subprocess.Popen(stacks_cmd, shell=True, stdout=subprocess.PIPE)
stacks_output, stacks_error = process.communicate()

That didn't work, and fails with the same error.

I've tried using Popen in shell=False mode

import subprocess
stacks_cmd = ['/Users/<username>/EMAN2/bin/python', '/Users/<username>/EMAN2/bin/e2buildstacks.py', <args>]
process = subprocess.Popen(stacks_cmd, shell=False, stdout=subprocess.PIPE)
stacks_output, stacks_error = process.communicate()

or 

import subprocess
stacks_cmd = ['/Users/<username>/EMAN2/bin/e2buildstacks.py', <args>]
process = subprocess.Popen(stacks_cmd, shell=False, stdout=subprocess.PIPE)
stacks_output, stacks_error = process.communicate()

Both of those fail with the same error.

I've tried specifying the executable:

import subprocess
stacks_cmd = ['/Users/<username>/EMAN2/bin/e2buildstacks.py', <args>]
process = subprocess.Popen(stacks_cmd, shell=False, stdout=subprocess.PIPE, executable='/Users/<username>/EMAN2/bin/python')
stacks_output, stacks_error = process.communicate()

This one gives a different error:

Unknown option: --
usage: /Users/<username>/EMAN2/bin/e2buildstacks.py [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.

Which makes me think that somehow the arguments aren't getting passed correctly to the script, but are rather being passed to python itself (I probably don't really understand what the executable setting does here).

I've tried setting the environment variable:

import os
my_env = os.environ.copy()
my_env['PATH'] = '/Users/<username>/EMAN2/bin:'+my_env['PATH']
my_env['PYTHONPATH'] = '/Users/<username>/EMAN2/bin'

And then passing that to subprocess.Popen in any of the above commands as env=my_env. This fails with the same errors.

At this point, I'm pretty much out of ideas and hoping someone can help. Again, this is only happening with pythonw, not python.

Things I've read looking for solutions

I couldn't find anything on stackoverflow that quite matched this problem. things I've looked at:

subprocess a program using another version of python

  • Never answered successfully.

Module cannot be found when using "pythonw" (instead of "python") to run an application

  • The problem is not which python/pythonw I'm using, both are coming from miniconda.

Python subprocess is running a different version of Python

  • This is kind of the exact opposite of my problem, so not much use.

https://github.com/ContinuumIO/anaconda-issues/issues/199

  • This suggests that the miniconda pythonw is a bit of a hack that can cause various problems, but doesn't directly offer a solution (lets say that using another version of python is strongly discouraged).

Python subprocess/Popen with a modified environment

Python Script not running in crontab calling pysaunter

  • These led me to trying to modify the environment variables.

Final note

  • Changing the python distribution being used is possible, but very much not the ideal solution. I'm writing something that will be run in several different environments (including linux and windows, which I haven't tested on yet) and by people who aren't me, so I need a more bulletproof solution.

Any help folks can provide is much appreciated.

jhopkins
  • 21
  • 3

0 Answers0