4

On macOS Sierra 10.12.6, environment variable LD_LIBRARY_PATH cannot be used by os.system(), subprocess.run() and subprocess.Popen(), even though PATH can be used normally. Python version is 3.6.1. But on Linux(Ubuntu 17.10), this environment variable can also be used rightly. The following python script envv.py can show this problem:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import subprocess
import os
import time

PATH = "PATH"
print(os.environ.get(PATH))
os.system("echo $" + PATH)
subprocess.run("echo $" + PATH, shell=True)
subprocess.run("/bin/echo $" + PATH, shell=True)
subprocess.run("/bin/echo $" + PATH, shell=True, env={PATH: os.environ.get(PATH)})
subprocess.run("/bin/echo $" + PATH, shell=True, env=os.environ)
subprocess.Popen("/bin/echo $" + PATH, shell=True, env=os.environ.copy())
time.sleep(2)
print('\n')

LD_LIBRARY_PATH = "LD_LIBRARY_PATH"
print(os.environ.get(LD_LIBRARY_PATH))
os.system("echo $" + LD_LIBRARY_PATH)
subprocess.run("/bin/echo $" + LD_LIBRARY_PATH, shell=True)
subprocess.run("/bin/echo $" + LD_LIBRARY_PATH, shell=True, env={LD_LIBRARY_PATH: os.environ.get(LD_LIBRARY_PATH)})
subprocess.run("/bin/echo $" + LD_LIBRARY_PATH, shell=True, env=os.environ)
subprocess.Popen("/bin/echo $" + LD_LIBRARY_PATH, shell=True, env=os.environ.copy())
time.sleep(2)
print('\n')

On macOS, the output is

$ python3 envv.py 
/opt/alps/bin:...
/opt/alps/bin:...
/opt/alps/bin:...
/opt/alps/bin:...
/opt/alps/bin:...
/opt/alps/bin:...
/opt/alps/bin:...


/opt/alps/lib:...
$    # use $ to show blank line
$
....

On Linux, the output is

$ python3 envv.py 
/opt/intel/compilers_and_libraries_2018.1.163/linux/bin/intel64:...
/opt/intel/compilers_and_libraries_2018.1.163/linux/bin/intel64:...
/opt/intel/compilers_and_libraries_2018.1.163/linux/bin/intel64:...
/opt/intel/compilers_and_libraries_2018.1.163/linux/bin/intel64:...
/opt/intel/compilers_and_libraries_2018.1.163/linux/bin/intel64:...
/opt/intel/compilers_and_libraries_2018.1.163/linux/bin/intel64:...
/opt/intel/compilers_and_libraries_2018.1.163/linux/bin/intel64:...


/opt/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64:...
/opt/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64:...
/opt/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64:...
/opt/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64:...
/opt/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64:...
/opt/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64:...

Does anyone have any idea? Thanks a lot!

UPDATE 2018-02-07

Follow @GrahamDumpleton 's suggestion, I tried DYLD_LIBRARY_PATH, but got the same result. The test stript is

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import subprocess
import os

DYLD_LIBRARY_PATH = "DYLD_LIBRARY_PATH"
print(os.environ.get(DYLD_LIBRARY_PATH))
os.system("echo $" + DYLD_LIBRARY_PATH)
subprocess.run("/bin/echo $" + DYLD_LIBRARY_PATH, shell=True)
subprocess.run("/bin/echo $" + DYLD_LIBRARY_PATH, shell=True, env={DYLD_LIBRARY_PATH: os.environ.get(DYLD_LIBRARY_PATH)})
subprocess.run("/bin/echo $" + DYLD_LIBRARY_PATH, shell=True, env=os.environ)
subprocess.Popen("/bin/echo $" + DYLD_LIBRARY_PATH, shell=True, env=os.environ.copy())

And the related output is

$ python3 envv.py
/opt/intel/compilers_and_libraries_2018.1.126/mac/compiler/lib:...
$
$
...
thorough
  • 63
  • 1
  • 5
  • On macOS for indicating where shared libraries can be loaded, it is ``DYLD_LIBRARY_PATH`` and not ``LD_LIBRARY_PATH`` for a start. So any behaviour seen shouldn't be specific to that variable name, so try with a different variable name. If ``DYLD_LIBRARY_PATH`` is used, there are some strange places where it doesn't get inherited on macOS, but you aren't using that name. – Graham Dumpleton Feb 07 '18 at 07:11
  • @GrahamDumpleton Thanks for your comment. I have tried `DYLD_LIBRARY_PATH` but got the same result as `LD_LIBRARY_PATH`. And I have updated the question. – thorough Feb 07 '18 at 12:37
  • Now read https://developer.apple.com/library/content/documentation/Security/Conceptual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtections.html#//apple_ref/doc/uid/TP40016462-CH3-SW1 and https://stackoverflow.com/questions/33616185/bash-export-not-working-for-only-one-variable I only expect it to apply to DYLD_* variables though, not LD_LIBRARY_PATH. – Graham Dumpleton Feb 07 '18 at 20:18
  • @GrahamDumpleton https://forums.developer.apple.com/thread/9233 is related to this problem. In practice, what I want is a python wrapper for a numerical simulation program which is written by Fortran and MKL library is used dynamically. I use `subprocess.run("FORTRAN_PROG.exe")` to run the program, so `DYLD_LIBRARY_PATH` is needed here. – thorough Feb 08 '18 at 03:18

1 Answers1

2

If you're using subprocess.Popen with shell=True, you can manually propagate the LD_LIBRARY_PATH and DYLD_LIBRARY_PATH variables like so:

import os
import sys
import subprocess

cmd = "/bin/echo $DYLD_LIBRARY_PATH"

sys_env = os.environ.copy()

# We've used os.environ.copy() so we can make mods
# to the subprocess environment if needed without
# affecting the parent process.

if sys.platform == 'darwin':
    if "LD_LIBRARY_PATH" in sys_env:
        cmd = f"export LD_LIBRARY_PATH={sys_env['LD_LIBRARY_PATH']} && {cmd}"
    if "DYLD_LIBRARY_PATH" in sys_env:
        cmd = f"export DYLD_LIBRARY_PATH={sys_env['DYLD_LIBRARY_PATH']} && {cmd}"

process = subprocess.Popen(
    cmd,
    env=sys_env,
    shell=True,
)
micah
  • 48
  • 5