Calling the "source" command from subprocess.Popen

Question

I have a .sh script that I call with source the_script.sh. Calling this regularly is fine. However, I am trying to call it from my python script, through subprocess.Popen.

Calling it from Popen, I am getting the following errors in the following two scenario calls:

foo = subprocess.Popen("source the_script.sh")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/subprocess.py", line 672, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1213, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory


>>> foo = subprocess.Popen("source the_script.sh", shell = True)
>>> /bin/sh: source: not found

What gives? Why can't I call "source" from Popen, when I can outside of python?

Answer 1

source is not an executable command, it's a shell builtin.

The most usual case for using source is to run a shell script that changes the environment and to retain that environment in the current shell. That's exactly how virtualenv works to modify the default python environment.

Creating a sub-process and using source in the subprocess probably won't do anything useful, it won't modify the environment of the parent process, none of the side-effects of using the sourced script will take place.

Python has an analogous command, execfile, which runs the specified file using the current python global namespace (or another, if you supply one), that you could use in a similar way as the bash command source.

Answer 2

You could just run the command in a subshell and use the results to update the current environment.

def shell_source(script):
    """Sometime you want to emulate the action of "source" in bash,
    settings some environment variables. Here is a way to do it."""
    import subprocess, os
    pipe = subprocess.Popen(". %s; env" % script, stdout=subprocess.PIPE, shell=True)
    output = pipe.communicate()[0]
    env = dict((line.split("=", 1) for line in output.splitlines()))
    os.environ.update(env)

Answer 3

Broken Popen("source the_script.sh") is equivalent to Popen(["source the_script.sh"]) that tries unsuccessfully to launch 'source the_script.sh' program. It can't find it, hence "No such file or directory" error.

Broken Popen("source the_script.sh", shell=True) fails because source is a bash builtin command (type help source in bash) but the default shell is /bin/sh that doesn't understand it (/bin/sh uses .). Assuming there could be other bash-ism in the_script.sh, it should be run using bash:

foo = Popen("source the_script.sh", shell=True, executable="/bin/bash")

As @IfLoop said, it is not very useful to execute source in a subprocess because it can't affect parent's environment.

os.environ.update(env) -based methods fail if the_script.sh executes unset for some variables. os.environ.clear() could be called to reset the environment:

#!/usr/bin/env python2
import os
from pprint import pprint
from subprocess import check_output

os.environ['a'] = 'a'*100
# POSIX: name shall not contain '=', value doesn't contain '\0'
output = check_output("source the_script.sh; env -0",   shell=True,
                      executable="/bin/bash")
# replace env
os.environ.clear() 
os.environ.update(line.partition('=')[::2] for line in output.split('\0'))
pprint(dict(os.environ)) #NOTE: only `export`ed envvars here

It uses env -0 and .split('\0') suggested by @unutbu

To support arbitrary bytes in os.environb, json module could be used (assuming we use Python version where "json.dumps not parsable by json.loads" issue is fixed):

To avoid passing the environment via pipes, the Python code could be changed to invoke itself in the subprocess environment e.g.:

#!/usr/bin/env python2
import os
import sys
from pipes import quote
from pprint import pprint

if "--child" in sys.argv: # executed in the child environment
    pprint(dict(os.environ))
else:
    python, script = quote(sys.executable), quote(sys.argv[0])
    os.execl("/bin/bash", "/bin/bash", "-c",
        "source the_script.sh; %s %s --child" % (python, script))

Answer 4

source is a bash-specific shell built-in (and non-interactive shells are often lightweight dash shells instead of bash). Instead, just call /bin/sh:

foo = subprocess.Popen(["/bin/sh", "the_script.sh"])

Answer 5

Update: 2019

"""
    Sometime you want to emulate the action of "source" in bash,
    settings some environment variables. Here is a way to do it.
"""
def shell_source( str_script, lst_filter ):
    #work around to allow variables with new lines
    #example MY_VAR='foo\n'
    #env -i create clean shell
    #bash -c run bash command
    #set -a optional include if you want to export both shell and enrivonment variables
    #env -0 seperates variables with null char instead of newline
    command = shlex.split(f"env -i bash -c 'set -a && source {str_script} && env -0'")

    pipe = subprocess.Popen( command, stdout=subprocess.PIPE )
    #pipe now outputs as byte, so turn it to utf string before parsing
    output = pipe.communicate()[0].decode('utf-8')
    #There was always a trailing empty line for me so im removing it. Delete this line if this is not happening for you.
    output = output[:-1]

    pre_filter_env = {}
    #split using null char
    for line in output.split('\x00'):
        line = line.split( '=', 1)
        pre_filter_env[ line[0]] = line[1]

    post_filter_env = {}
    for str_var in lst_filter:
        post_filter_env[ str_var ] = pre_filter_env[ str_var ]

    os.environ.update( post_filter_env )

Answer 6

A variation on the answer by @xApple since it sometimes is useful to be able to source a shell script (rather than a Python file) to set environment variables, and maybe perform other shell operations, and then propagate that environment to the Python interpreter rather than losing that info when the subshell closes.

The reason for a variation is that the assumption of a one-variable-per-line format of output from "env" isn't 100% robust: I just had to deal with a variable (a shell function, I think) containing a newline, which screwed up that parsing. So here is a slightly more complex version, which uses Python itself to format the environment dictionary in a robust way:

import subprocess
pipe = subprocess.Popen(". ./shellscript.sh; python -c 'import os; print \"newenv = %r\" % os.environ'", 
    stdout=subprocess.PIPE, shell=True)
exec(pipe.communicate()[0])
os.environ.update(newenv)

Maybe there is a neater way? This also ensures that the environment parsing isn't messed up if someone puts an echo statement into the script that's being sourced. Of course, there's an exec in here so be careful about non-trusted input... but I think that's implicit in a discussion about how to source/execute an arbitrary shell script ;-)

UPDATE: see @unutbu's comment on the @xApple answer for an alternative (probably nicer) way to handle newlines in the env output.

Answer 7

Using the answers here I created a solution that fit my needs.

• no need to filter out env variables
• allows for variables with new line characters

def shell_source(script):
    """
    Sometime you want to emulate the action of "source" in bash,
    settings some environment variables. Here is a way to do it.
    """
    
    pipe = subprocess.Popen(". %s && env -0" % script, stdout=subprocess.PIPE, shell=True)
    output = pipe.communicate()[0].decode('utf-8')
    output = output[:-1] # fix for index out for range in 'env[ line[0] ] = line[1]'

    env = {}
    # split using null char
    for line in output.split('\x00'):
        line = line.split( '=', 1)
        # print(line)
        env[ line[0] ] = line[1]

    os.environ.update(env)

With this I was able to run commands with the same environment variables without issue:

def runCommand(command):
    """
    Prints and then runs shell command.
    """
    print(f'> running: {command}')
    stream = subprocess.Popen(command, shell=True,env=os.environ)
    (result_data, result_error) = stream.communicate()
    print(f'{result_data}, {result_error}')

Hope this helps anyone in the same position as me

Answer 8

If you want to apply source command to some other scripts or executables, then you might create another wrapping script file and call "source" command from it with any further logic you need. In that case, this source command will modify local context where it running - namely in the subprocess that subprocess.Popen creates.

This will not work if you need to modify python context, where your program is running.

Answer 9

There seem to be a lot of answers to this, haven't read all of them so they may have already pointed it out; but, when calling shell commands like this, you have to pass shell=True to the Popen call. Otherwise, you can call Popen(shlex.split()). make sure to import shlex.

I actually use this function for the purpose of sourcing a file and modifying the current environment.

def set_env(env_file):
    while True:
        source_file = '/tmp/regr.source.%d'%random.randint(0, (2**32)-1)
        if not os.path.isfile(source_file): break
    with open(source_file, 'w') as src_file:
        src_file.write('#!/bin/bash\n')
        src_file.write('source %s\n'%env_file)
        src_file.write('env\n')
    os.chmod(source_file, 0755)
    p = subprocess.Popen(source_file, shell=True,
                         stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    (out, err) = p.communicate()
    setting = re.compile('^(?P<setting>[^=]*)=')
    value = re.compile('=(?P<value>.*$)')
    env_dict = {}
    for line in out.splitlines():
        if setting.search(line) and value.search(line):
            env_dict[setting.search(line).group('setting')] = value.search(line).group('value')
    for k, v in env_dict.items():
        os.environ[k] = v
    for k, v in env_dict.items():
        try:
            assert(os.getenv(k) == v)
        except AssertionError:
            raise Exception('Unable to modify environment')