How to get environment variables after running a subprocess

Question

I am using the subprocess.call to execute a shell script from another application I am integrating with. This script sets environment variables with export MY_VAR=foo. Next, I need to execute more commands over subprocess with the environment that was set by the shell script.

How to extract the state of environment from the child process? It only returns the errno code.

i.e. I want to run:

subprocess.call(["export", "MY_VAR=foo"]
subprocess.call(["echo", "$MY_VAR"])  # should print 'foo'.

I know that I can set environment with env keyword, but the point of my question is how to get the environment variables that a subprocess sets. In shell you can source any script to get it's declared environment variables. What's the alternative in python?

Answer 1

I ran into this issue just recently. It seems that this is a difficult problem for reasons upstream of Python: posix_spawn doesn't give a way to read the environment variables of the spawned process, nor is there any easy way to read the environment of a running process.

Bash's source is specific to running bash code in the bash interpreter: it just evals the file in the current bash interpreter rather than starting a subprocess. This mechanism can't work if you are running bash code from Python.

It is possible to make a separate mechanism specific to running bash code from Python. The following is the best that I could manage. Would be nice to have a less flimsy solution.

import json
import os
import subprocess
import sys

from contextlib import AbstractContextManager


class BashRunnerWithSharedEnvironment(AbstractContextManager):
    """Run multiple bash scripts with persisent environment.

    Environment is stored to "env" member between runs. This can be updated
    directly to adjust the environment, or read to get variables.
    """

    def __init__(self, env=None):
        if env is None:
            env = dict(os.environ)
        self.env: Dict[str, str] = env
        self._fd_read, self._fd_write = os.pipe()

    def run(self, cmd, **opts):
        if self._fd_read is None:
            raise RuntimeError("BashRunner is already closed")
        write_env_pycode = ";".join(
            [
                "import os",
                "import json",
                f"os.write({self._fd_write}, json.dumps(dict(os.environ)).encode())",
            ]
        )
        write_env_shell_cmd = f"{sys.executable} -c '{write_env_pycode}'"
        cmd += "\n" + write_env_shell_cmd
        result = subprocess.run(
            ["bash", "-ce", cmd], pass_fds=[self._fd_write], env=self.env, **opts
        )
        self.env = json.loads(os.read(self._fd_read, 5000).decode())
        return result

    def __exit__(self, exc_type, exc_value, traceback):
        if self._fd_read:
            os.close(self._fd_read)
            os.close(self._fd_write)
            self._fd_read = None
            self._fd_write = None
    
    def __del__(self):
        self.__exit__(None, None, None)

Example:

with BashRunnerWithSharedEnvironment() as bash_runner:
    bash_runner.env.pop("A", None)

    res = bash_runner.run("A=6; echo $A", stdout=subprocess.PIPE)
    assert res.stdout == b'6\n'
    assert bash_runner.env.get("A", None) is None

    bash_runner.run("export A=2")
    assert bash_runner.env["A"] == "2"

    res = bash_runner.run("echo $A", stdout=subprocess.PIPE)
    assert res.stdout == b'2\n'

    res = bash_runner.run("A=6; echo $A", stdout=subprocess.PIPE)
    assert res.stdout == b'6\n'
    assert bash_runner.env.get("A", None) == "6"


    bash_runner.env["A"] = "7"
    res = bash_runner.run("echo $A", stdout=subprocess.PIPE)
    assert res.stdout == b'7\n'
    assert bash_runner.env["A"] == "7"

Answer 2

It is not possible, because the environment is changed only in the child process. You might from there return it as output to STDOUT, STDERR - but as soon as the subprocess is terminated, You can not access anything from it.

# this is process #1
subprocess.call(["export", "MY_VAR=foo"]

# this is process #2 - it can not see the environment of process #1
subprocess.call(["echo", "$MY_VAR"])  # should print 'foo'.

Answer 3

Not sure I see the problem here. You just need to remember the following:

• each subprocess that gets started is independent of any setups done in previous subprocesses
• if you want to set up some variables and use them, do both those things in ONE process

---

So make setupVars.sh like this:

export vHello="hello"
export vDate=$(date)
export vRandom=$RANDOM

And make printVars.sh like this:

#!/bin/bash
echo $vHello, $vDate, $vRandom

And make that executable with:

chmod +x printVars.sh

---

Now your Python looks like this:

import subprocess

subprocess.call(["bash","-c","source setupVars.sh; ./printVars.sh"])

Output

hello, Mon Jul 12 00:32:29 BST 2021, 8615

Answer 4

A really simple solution:

1. Save the first process envvars in a file with something like first-process-command && env > /tmp/env.txt.
2. Get the envvars from the file and pass them to the second process.

Working example (Python version >= 3.6) with three chained processes:

import os
import subprocess


OUTBOUND_ENVVARS_FILEPATH = '/tmp/env.txt'


def run(command, env=None):
    env = env if env is not None else {}
    process = subprocess.Popen(
        f'{command} && env > {OUTBOUND_ENVVARS_FILEPATH}',
        env=env,
        shell=True)
    process.wait()
    return get_outbound_envvars()


def get_outbound_envvars():
    env = {}

    with open(OUTBOUND_ENVVARS_FILEPATH, 'r') as outbound_envvars_file:
        for line in outbound_envvars_file.readlines():
            parts = line.strip().split('=')
            key = parts[0]
            value = parts[1]
            env[key] = value
    
    os.remove(OUTBOUND_ENVVARS_FILEPATH)

    return env


shared_env = run('export MY_VAR_1="world" && echo -n "hello "')
shared_env = run('export MY_VAR_2="!" && echo -n $MY_VAR_1', shared_env)
run('echo $MY_VAR_2', shared_env)

Output: hello world!.

Notes:

• Notice the process.wait() and shell=True lines.
• Maybe you want to remove some envvars like PWD or OLDPWD from env dict, if they're present.