0

I am trying to run a linux executable on Max OS X 10.11.6 via python2.7

I would like to use subprocess.check_output.

The command, which works via the terminal is:

mosel -c "exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"

However, when I try:

subprocess.check_output(['mosel','-c',cmd])

where

cmd="exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"'

I get:

File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 629, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 825, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 1574, in _execute_child
raise child_exception_type(errno_num, err_msg)
OSError: [Errno 2] No such file or directory: 'mosel'

I have been able to get it to "echo" the command to an output file, but I cannot run "which mosel" via python, which leads me to believe that it has to do with check_output using "bin/sh"as the executable.

So, do I need to use "Popen" instead and set 

executable=path/to/mosel

?

If so, how do use Python to get the user's path to mosel (i.e. get the output of "which mosel")?

Thanks!

UPDATE:

PyCharm was not seeing the system paths, which I fixed using this answer: Setting environment variables in OS X?

Now, it appears that 

subprocess.check_output(['mosel','-c',cmd])

Is sending the square brackets to the command line, because it now returns:

dyld: Library not loaded: libxprm_mc.dylib
Referenced from: /usr/local/opt/xpress/bin/mosel
Reason: image not found
Traceback (most recent call last):
File "/Users/nlaws/projects/sunlamp/sunlamp-ss/RunScenarios/run.py", line 70, in <module>
run(1)
File "/Users/nlaws/projects/sunlamp/sunlamp-ss/RunScenarios/run.py", line 44, in run
out = check_output(['mosel', '-c', cmd])
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 219, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['mosel', '-c', cmd]' returned non-zero exit status -5

Or is there still a path issue?! (I can run mosel -c cmd via the mac terminal, but not in pycharm via python, nor in the mac terminal via python).

Community
  • 1
  • 1
Nick Laws
  • 15
  • 1
  • 6
  • 2
    When `shell=False` you *have* to pass a list of string. so the last example is correct. However you did not provide any information about what happened in that case. – Bakuriu Jan 05 '17 at 16:49
  • The same thing happens as the first output – Nick Laws Jan 05 '17 at 16:59
  • Well, `mosel` is *not* correctly installed in your system. Fix your installation. – Bakuriu Jan 05 '17 at 17:18
  • Also: `check_output` does **not** use any shell (by default)! so what you are saying about `/bin/sh` is irrelevant. `check_output` searchs the executable with that name in the current `PATH` and launches it. The error simply means that it cannot find the file. In other words: the python code is fine, the problem is in your path. Which was your current directory when launching that executable from command line and from python? Where is this mosel executable located? (In a user directory, or in a system directory together with other standard executables?) – Bakuriu Jan 05 '17 at 17:21
  • @Bakuriu I believe that mosel is correctly installed because when I enter the exact same line: 'mosel -c "exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"` in the terminal everything works fine. I have checked all of the path issues, and I have run the mosel command in the terminal from the same directory as the python script. – Nick Laws Jan 05 '17 at 17:22
  • @Bakuriu Do you know how to execute "which mosel" via python and grab the output? I believe that the solution to that problem is the same as how to execute "mosel" via python. – Nick Laws Jan 05 '17 at 17:29
  • `which mosel` will not help you, because if Python wasn't able to find `mosel`, `which` wont too! `which` simply searches the `PATH` for the executable, exactly like what you are doing in python. – Bakuriu Jan 05 '17 at 17:47
  • @Bakuriu `which` does find `mosel`. I believe that it is not a path issue. – Nick Laws Jan 05 '17 at 18:19
  • No, it does not. **When you launch it from the shell it does**, but it wouldn't find it if you launched from python, otherwise your original command would work. Hear me: either you are saying that python is bugged, and so you should close this question and open a ticket in python's issue tracker, or you are screwing with the environment in some way( and you didn't provide us a way to reproduce your environment, hence we are helpless here). – Bakuriu Jan 05 '17 at 18:57
  • @Bakuriu I see what you are saying: `os.getenv('PATH')` returns: `'/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin'` And in terminal: `echo $PATH` returns: `/usr/local/opt/xpress/bin:`.... How do I get python to see the system PATH? – Nick Laws Jan 05 '17 at 19:32

2 Answers2

1

The problem is you're using check_output's arguments incorrectly. Unless you pass it shell=True, check_output expects a list of parameters as its input, in the form: check_call(['binary_name','arg1','arg2','arg3', ....])

So in this case, you should do:

subprocess.check_call(['mosel', '-c', "exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"])
aruisdante
  • 8,875
  • 2
  • 30
  • 37
  • This method results in: `OSError: [Errno 2] No such file or directory: 'mosel'` – Nick Laws Jan 05 '17 at 17:02
  • @NickLaws please update the original question with the output from trying this form if you've done it, it will be clearer to future readers. – aruisdante Jan 05 '17 at 17:03
0

The root of the issue turns out to be the DYLD_LIBRARY_PATH:

The new OS X release 10.11 "El Capitan" has a "security" feature that prevents passing DYLD_LIBRARY_PATH to child processes. Somehow, that variable is stripped from the environment. - (quoted from https://www.postgresql.org/message-id/20151103113612.GW11897@awork2.anarazel.de)

The security feature is called SIP or "System Integrity Protection". Unfortunately, it seems that no one has come up with a solution to this issue (other than work-arounds that must be tailored to each situation).

Here is another example of this issue: https://github.com/soumith/cudnn.torch/issues/111

Google "mac os inherit dyld_library_path" and you will find many other examples.

Nick Laws
  • 15
  • 1
  • 6