How to set a thread specific environment variable in Python?

Question

I want to create two (or more) threads and in each of them to execute a different external program, let's say aaa and bbb. These external programs need libraries located in different directories, let's say in /aaalib and /bbblib, so I have to set the environment variable LD_LIBRARY_PATH before executing them. I want to avoid using shell, so solutions like execution LD_LIBRARY_PATH=/aaalib aaa ain't good.

The question is: how do I set the os.environ['LD_LIBRARY_PATH'] in such a way that it will have different values in different threads?

PS. I tried the solution os.environ['...'] = threading.local() described here, but it fails in following way:

In [1]: import os, threading
In [2]: os.environ['LD_LIBRARY_PATH'] = threading.local()
-----------------------
TypeError           Traceback (most recent call last)
<ipython-input-2-a2c8ef0b901b> in <module>()
----> 1 os.environ['LD_LIBRARY_PATH'] = threading.local()
/usr/lib/python2.7/os.pyc in __setitem__(self, key, item)
471                 self.data = environ
472             def __setitem__(self, key, item):
--> 473                 putenv(key, item)
474                 self.data[key] = item
475             def update(self,  dict=None, **kwargs):
TypeError: must be string, not thread._local

Answer 1

you can try using subprocess module and do something like this:

import subprocess
import os

env = os.environ.copy()
env['LD_LIBRARY_PATH'] = '/aaalib'
aaa_process = subprocess.Popen(['aaa'], env=env)

env = os.environ.copy()
env['LD_LIBRARY_PATH'] = '/bbblib'
bbb_process = subprocess.Popen(['bbb'], env=env)

Answer 2

First of all I guess threads stay in the same environment so I advice you to use multiprocessing or subprocess library to handle processes and not threads. in each process function you can change the environment freely and independently from the parent script

each process should have this kind of function

def target(some_value):
    os.environ['FOO'] = some_value
    # some_code_here

some_value to be passed through the spawn of the process

p = multiprocessing.Process(name=,target=target,args=(some_value,))
p.start()

Answer 3

Using the wonderful plumbum library:

from plumbum import local
with local.env(LD_LIBRARY_PATH = '/aaalib'):
    execute_external_program()

See docs.

Note that you should also use plumbum to execute_external_program, meaning don't use subprocess (or its alternatives) directly. For example, to test this env-setting logic, you can do:

from plumbum import local
with local.env(LD_LIBRARY_PATH = '/aaalib'):
    print(local.python("-c", "import os;print os.environ['LD_LIBRARY_PATH']"))

To be clear, this solution guarantees that the commands you run in subprocesses (using plumbum) see the env you want, and you don't need to modify parent-process's env vars to achieve that, and you don't need to use the "raw" subprocess module directly.

Answer 4

A threads cannot have its own environment variables, it always has those of its parent process. You can either set the appropriate values when you create the subprocesses for your external programs or use separate processes instead of threads.