1

I had a very strange experience with env variables.

In summary, I needed to set a env variable VAR1, I was 99% sure I ran the command export VAR1=some-value. However, after a few hours I forgot whether I set it or not, so I ran echo $VAR1 to check. And the output is exactly some-value, which confirmed that I set it correctly.

However, when I do this in python:

import os
print("VAR1" in os.environ)

The output is False.

I was very confused at this point. If I trust my python output, it means that my way of checking a env variable using echo was wrong. Is that the case?

Because I don't know what's wrong, I cannot provide a reproducible code sample. I really appreciate any explanation.

Hongtao Yang
  • 381
  • 3
  • 14
  • 2
    Could it be that you set it but didn't export it? That would explain it. – CryptoFool Nov 02 '20 at 05:43
  • That could be the case. So what's the correct way to check for a env variable? – Hongtao Yang Nov 02 '20 at 05:45
  • Might also be that `export` exports the env variable in the child process of the shell where it was ran. – BernardL Nov 02 '20 at 05:47
  • Your way to check if a var is set is fine. To see if it is exported, use `export | grep `. See the bottom of my answer, where I added this information. – CryptoFool Nov 02 '20 at 05:50
  • Do you understand what it means to `export` a variable? It means that it will be inherited by child processes. So when you launch Python from the command line, that launches a child process of the current shell process. Only exported variables are copied into the child process, and so only exported variables are seen by Python – CryptoFool Nov 02 '20 at 05:54
  • Thanks, I did not understand what `export` means. I really appreciate your explaination! – Hongtao Yang Nov 02 '20 at 05:56
  • Coolio! It's a good thing to understand, as I say in my answer :o) – CryptoFool Nov 02 '20 at 06:04

1 Answers1

6

The only thing I can think of is that you were mistaken in that you set it but did not export it. That would explain why you could echo the value, but it didn't make into os.environ in Python. Here's a demonstration:

>>> VAR1="vvv1"
>>> export VAR2="vvv2"
>>> echo $VAR1
vvv1
>>> echo $VAR2
vvv2
>>> python
Python 3.7.3 (default, Sep 16 2020, 12:18:14)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> "VAR1" in os.environ
False
>>> "VAR2" in os.environ
True

So VAR1 was only set, but VAR2 was set and exported. Both can be echoed, but only the exported one shows up in Python.

To check at the command line for if a variable is set and exported, use export and grep:

>>> export | grep VAR1
>>> export | grep VAR2
declare -x VAR2="vvv2"
>>>

It's helpful to really understand what export does. Only an exported variable will be inherited by child processes launched by the current shell process. So when you launch Python from the command line, that launches a child process. Only exported variables are copied into the child process, and so only exported variables are seen by Python.

It is this child process thing that explains why you can't run a shell script to set environment variables in your current shell. Because that script will run in a child process, it will only affect the variables in that child process. Once the script runs, that child process goes away. The main process's variable space was not affected by the script.

There is a way to run a script to set environment variables, and it can also be used to run a script that will have access to unexported variables. That is to run a script with '.' or 'source'. When you do . myscript.sh or source myscript.sh, this causes your script to be run in the current shell process. It prevents a subprocess from launching to run the script. So then the script is seeing and affecting the main shell environment.

Another small bit of trivia. I wasn't sure if there was any difference between . myscript.sh and source myscript.sh. Per this SO question, the only difference is portability. In Bash and other modern shells, there is no difference, but not all shells support both variants.

CryptoFool
  • 21,719
  • 5
  • 26
  • 44
  • Thanks so much for your share of knowledge. A follow up question, what happens when you set a var this way: `VAR=some-value python xxx.py`? Here no export is needed. – Hongtao Yang Nov 02 '20 at 06:06
  • Ah. Good question. That sets VAR only in the child process that's about to be launched. - so as you've probably seen, that's basically a way to set the variable just for that program you're running. – CryptoFool Nov 02 '20 at 06:09