1

Here's one of those rare questions that gets asked a lot on SO, but there's no answer. I'll ask it again a little differently.

I'm trying to set an environment variable using a file and then read the variable in python. That should be pretty basic.

However, check this out:

$ cat .env.test
NAME=Bob
$ cat x.py
import os
print(os.environ.get("NAME"))
$ (source .env.test;  echo $NAME) 
Bob
$ (source .env.test;  python3 x.py)
None

Isn't that strange? Why doesn't it work?

Is it because I didn't write export in the .env file? If I add export, it does work. But why? Isn't this what source is supposed to do?

$ cat .env.test
export NAME=Bob
$ cat x.py
import os
print(os.environ.get("NAME"))
$ (source .env.test;  echo $NAME) 
Bob
$ (source .env.test;  python3 x.py)
Bob

I think the community could use an explanation about how sub-shells work, how source works and where python is looking for the environment variables, to answer this once and for all. Can you do that?

Related Questions

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
101010
  • 14,866
  • 30
  • 95
  • 172
  • `export` is what adds the variable to the environment. In your first example, you only have a shell variable. – chepner Sep 24 '22 at 14:36
  • This really has nothing to do with Python, but with understanding how your particular dshell works. – chepner Sep 24 '22 at 14:37
  • you don't define a env var, just a var. – azro Sep 24 '22 at 14:39
  • _reviews the related questions to see if they're really based on the same misapprehension_ – Charles Duffy Sep 24 '22 at 14:40
  • ...so, the claimed preexisting question *os.environ.get not returning value* is ambiguous -- we don't have enough details to reproduce the problem, so we can't tell if it's the same shell-vs-env dichotomy; so it's not in any way clear that it's really a duplicate at all. – Charles Duffy Sep 24 '22 at 14:41
  • The flask one, by contrast, has `sudo` in the way. That completely changes things, because sudo is often configured to discard environment variables. And in fact, the OP acknowledges in the comments on the question that that's their problem -- so it's completely distinct from this one. – Charles Duffy Sep 24 '22 at 14:42
  • And the first one you listed doesn't make it clear that the OP there understands that they need to start Python _from the same shell_ where they ran the `export`. – Charles Duffy Sep 24 '22 at 14:45
  • ...in fact, the OP there is clear that they're starting Python _from their IDE_, not from the shell; so that too is distinct. – Charles Duffy Sep 24 '22 at 14:45
  • ...I've added community-wiki answers to the two of those questions that have enough information to be answerable, and added an appropriate close vote to the third. – Charles Duffy Sep 24 '22 at 14:47

1 Answers1

1

var=value does not define an environment variable unless the option set -a is defined. By default, it only defines a shell variable.

Non-exported shell variables do not survive an exec() boundary -- that is to say, when you start a new executable, only environment variables survive.

To export all variables sourced in from a file, enable the set -a shell option before sourcing it, as described in How to set environment variables from .env file:

$ (set -a; source .env.test; exec python3 x.py)

In the above case we don't need to turn off set -a because it's scoped to a subshell that exits when the ) is hit -- or even before, because we can use exec to tell the subshell to replace itself with Python without forking again.

For a more general case, when this is used inside a script that's going to continue to run for a while, it would instead look like:

set -a            # enable auto-export
source .env.test  # source file content
set +a            # disable auto-export
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • Ah, shell variables vs environment-variable. Got it. Follow-up question: In this SO question, I'm trying to accomplish this with a file, can `source` be used in some way to implicitly create `export`'d environment variables? – 101010 Sep 24 '22 at 15:15
  • `(export $(cat .env.test | xargs); python3 x.py)` seems to do it – 101010 Sep 24 '22 at 15:19
  • 1
    That's buggy -- I don't recommend it. Better to use `set -a; source .env.test; set +a; python3 x.py`. – Charles Duffy Sep 24 '22 at 15:34
  • @101010, ...to go beyond a bare assertion re: "that's buggy", try `var='value with=multiple words=contained'` -- with the `export $(cat | xargs)` approach you'll see `var`, `with`, and `words` as three separate environment variables. – Charles Duffy Sep 24 '22 at 15:46
  • @101010, ...for an even more, err, _interesting_ case, try `var=$'value\nwith=newlines\nbetween=words'` -- that'll break a lot of other xargs-based approaches (when run in bash, that command results in a variable with newlines in it, not backslashes or `n` characters between the words). Also test with regular single quotes and literal newlines instead of `\n` sequences, when you're trying to find a suitably robust approach. – Charles Duffy Sep 24 '22 at 16:04
  • The one-liner is a great solution to this SO article. Thank you for the follow-up explanation. I added the one-liner to your answer. – 101010 Sep 25 '22 at 11:30
  • @101010, note that in the subshell case some additional optimizations are possible. You don't need `set +a` if the subshell is about to be destroyed, and you can use `exec` to tell the shell to just replace the subshell process with Python instead of forking off Python as a child -- which means you end up not paying any extra cost for the subshell at all, because the fork that starts the subshell is paid off by avoiding the fork that would otherwise be used to create a process in which to start Python. – Charles Duffy Sep 25 '22 at 13:26