How to set Environment Variables on EC2 instance via User Data

Question

I am trying to set environment variables with EC2s user data, but nothing i do seems to work

here are the User data scripts i tried

#!/bin/bash
echo "export HOST_URL=checkEmai-LoadBala-ICHJ82KG5C7P-23235232.us-east-1.elb.amazonaws.com" >> /env.sh 
source /env.sh

And another:

#!/bin/bash
echo "#!/bin/bash" >> /env.sh
echo "export HOST_URL=checkEmai-LoadBala-ICHJ82KG5C7P-67323523.us-east-1.elb.amazonaws.com" >> /env.sh 
chmod +x /env.sh
/env.sh

They both do absolutly nothing, and if i log in and issue the command source /env.sh or /env.sh it works. so this must be something forbidden that i am trying to do.

Here is the output from /var/log/cloud-init-output.log using -e -x

+ echo 'export HOST_URL=checkEmai-LoadBala-ICHJ82KG5C7P-2141709021.us-east-1.elb.amazonaws.com'
+ source /env.sh
++ export HOST_URL=checkEmai-LoadBala-ICHJ82KG5C7P-2141709022.us-east-1.elb.amazonaws.com
++ HOST_URL=checkEmai-LoadBala-ICHJ82KG5C7P-2141709022.us-east-1.elb.amazonaws.com

Still, echo $HOST_URL is empty

As requested, the full UserData script

#!/bin/bash
set -e -x 
echo "export HOST_URL=checkEmai-LoadBala-ICHJ82KG5C7P-2141709021.us-east-1.elb.amazonaws.com" >> /env.sh 
source /env.sh
/startup.sh staging 2649

Are you trying to set the environment variables for scripts that run later in the user-data, or do you want the variables available when you login to the server over SSH? Also, are you sure the AMI you are using supports user-data scripts? — Mark B, Dec 10 '15 at 15:23
1: i am trying to set user data to an application that runs after that script. 2: yes user data scripts work, this is just an addition to a script that runs after that one — Gleeb, Dec 10 '15 at 15:24
To be clear, the script you want the variables to be available to is also being called in your user-data script? — Mark B, Dec 10 '15 at 15:26
Can you show the entire user-data script then? Where are you running `echo $HOST_URL`? — Mark B, Dec 10 '15 at 15:47
running it after i log in using ssh, i'll paste the entire script — Gleeb, Dec 10 '15 at 15:47
See, I asked if you needed it in the user-data or later when you are logging in and you said you needed it in the user-data. It will be available later in the user-data script but not when you login via SSH. You would have to run `source /env.sh` after logging in, or add that command to your `.bashrc`, in order for it to be available when you login. — Mark B, Dec 10 '15 at 15:50
Ok sorry i misunderstood you. so the env variable HOST_URL should be available for everything that runs in the context of that script (meaning the startup.sh that i run after the source execution)? — Gleeb, Dec 10 '15 at 15:53
I should note that a point of having secrests as env variables is the security. If someone finds their way into the system, they'll be on a separate process and won't see all the secrets. Putting them in user data and pulling them to bashrc, as suggested by many solutions, increases the security risk. — anishtain4, Jul 11 '22 at 13:30

score 36 · Answer 1 · edited Oct 02 '20 at 03:26

36

One of the more configurable approach to define environment variables for EC2 instances, is to use Systems Manager Parameter Store. This approach will make it easier to manage different parameters for large number of EC2 instances, both encrypted using AWS KMS as well as in plain text. It will also allows to change the parameter values with minimal changes in EC2 instance level. The steps are as follows.

Define string parameters (Encrypted with KMS or Unencrypted) in EC2 Systems Manager Parameter Store.
In the IAM role EC2 assumes, give required permission to access the parameter store.
Using the AWS CLI commands for EC2 System Manager, read the parameters and export to environment variables in User Data section using Get-Parameter or Get-Parameters AWS CLI commands and controlling command output as required.

e.g Using Get-Parameter command to retrieve db_connection_string parameter(Unencrypted).

export DB_CONNECTION=$(aws --region=us-east-2 ssm get-parameter --name 'db_connection' --query 'Value')

Note: For more details in setting up AWS KMS Keys, defining encrypted strings, managing IAM policies & etc., refer the following articles.

edited Oct 02 '20 at 03:26

cussiol

7
3

answered Sep 17 '17 at 10:27

Ashan

18,898
4
47
67

The above script gives me null. Following worked for me ```export DB_CONNECTION=$(aws --region=ap-southeast-1 ssm get-parameters --names 'db_connection' --query "Parameters[0].Value")``` – Farhan Haider Jun 02 '21 at 02:41
Similar to the above comment `--query 'Parameter.Value'` is what worked for me. If you run the command without the `--query` you should see the parameter structure, and thru that you should be able to figure out what exactly you need to query. – Alex Jul 20 '21 at 14:46
1

Also, if you've stored your e.g. database URL as a `SecureString` with KMS, then append `--with-decryption` to the `aws ssm` command above in order to get that value decrypted. – Alex Jul 20 '21 at 14:49
You also need to trim off the `"` characters unless you are handling that in your application code. Piping that command to `tr -d \"` did the trick for me. – Sam Oct 25 '21 at 12:55
@sam , you can use `--output text` to trim the quotes – Stefan Bracke Mar 28 '22 at 12:18
1

You're the real MVP @stefan – Sam Jul 06 '22 at 12:55
This example does not work for me. I ended up using the other answer, ubuntu EC2 instance: `#!/bin/bash` `echo export DB_CONNECTION=$(aws ssm get-parameter --name 'DB_CONNECTION_SECRET' --query 'Parameter.Value' --with-decryption --region eu-west-1 --output text) >> /etc/profile` – JStanton Aug 19 '23 at 22:10

score 25 · Answer 2 · answered Apr 27 '17 at 18:18

25

I find this to be a pretty easy way to set environment variables for all users using User Data. It allows me to configure applications so the same AMI can work with multiple scenarios:

#!/bin/bash
echo export DB_CONNECTION="some DB connection" >> /etc/profile
echo export DB_USERNAME="my user" >> /etc/profile
echo export DB_PASSWORD="my password" >> /etc/profile

Now, all users will have DB_CONNECTION, DB_USERNAME and DB_PASSWORD set as environment variables.

answered Apr 27 '17 at 18:18

hamx0r

4,081
1
33
46

7

Does not work for me. I got the error: `-bash: /etc/profile: Permission denied`. PS: I try with sudo too. – Diego Somar Jun 25 '18 at 00:34
3

Good approach...I found that a little bit cleaner syntax on what gets echoed within the quotes worked perfectly. `echo "export GOROOT=/usr/local/go" >> /etc/profile` – ZacSketches Nov 27 '19 at 00:14
4

@DiegoSomar echo doesn't work with sudo. Use tee instead: `echo export DB_CONNECTION="some DB connection" | sudo tee -a /etc/profile` – Micah Henning Dec 01 '20 at 01:30
Best solution so far. – Nikhil Thakur Jul 10 '23 at 19:10
@DiegoSomar won't work with sudo, but you can use command "sudo su" to go to the root user then "echo export DB_CONNECTION="some DB connection" >> /etc/profile". It will work. – Nikhil Thakur Jul 10 '23 at 19:11

score 19 · Answer 3 · edited Nov 17 '20 at 08:56

The user data script on EC2 executes at after boot in its own process. The environment variables get set in that process and disappear when the process exits. You will not see the environment variables in other processes, i.e., login shell or other programs for that matter.

You will have to devise a way to get these environment variables into whatever program needs to see them.

Where do you need these variables to be available? In /startup.sh staging 2649?

EDIT

Try this:

#!/bin/bash
set -e -x 
export HOST_URL="checkEmai-LoadBala-ICHJ82KG5C7P-2141709021.us-east-1.elb.amazonaws.com"
/startup.sh staging 2649

Then edit /startup.sh, and put the following line on the top:

echo $HOST_URL > /tmp/var

Boot the instance, and then paste /tmp/var here.

the startup.sh runs a rails application, inside that rails application there is a use of that env variable — Gleeb, Dec 10 '15 at 16:04

score 7 · Answer 4 · edited Sep 17 '17 at 09:13

7

You can add another shell script in /etc/profile.d/yourscript.sh which will contain the set of environment variables you want to add.

This script will run at every bootup and your variable will be available to all users.

#!/bin/sh
echo 'export AWS_DEFAULT_REGION=ap-southeast-2' > ~/myconfiguration.sh
chmod +x ~/myconfiguration.sh
sudo cp ~/myconfiguration.sh /etc/profile.d/myconfiguration.sh

The above code creates a shell script to set environment variable for aws default region and copies it to profile.d .

edited Sep 17 '17 at 09:13

Phani Kumar M

4,564
1
14
26

answered Sep 17 '17 at 06:49

hardik khandelwal

96
1
2

After adding this, refresh the bash shell, command: `source /etc/profile` – Gowtham Chand Apr 08 '19 at 04:23

Avinash Sidhwani · Answer 5 · 2019-08-09T07:04:33.033

4

You can use this script:

#!/bin/bash
echo HOST_URL=\"checkEmai-LoadBala-ICHJ82KG5C7P-23235232.us-east-1.elb.amazonaws.com\" >> /etc/environment

I created an EC2 instance with Amazon Linux AMI 2018.03.0 and added this user data to it and it works fine.

Refer to this answer for more details.

edited Aug 09 '19 at 07:04

answered Aug 08 '19 at 19:23

Avinash Sidhwani

51
2

Seems like the only answer here that takes into account that `export` should not be in `/etc/environment` in (and maybe `/etc/profile` too) according to https://askubuntu.com/a/58828/615301 – Christiaan Westerbeek Sep 22 '20 at 07:04

score 2 · Answer 6 · answered Apr 06 '17 at 11:11

2

After doing the stuffs in the user data script, the process exits. So, whatever environment variable you export will not be there in the next process. One way is to to put exports in the .bashrc file so that it gets available in the next session also.

    echo "export HOST_URL=checkEmai-LoadBala-ICHJ82KG5C7P-23235232.us-east-1.elb.amazonaws.com" >> ~/.bashrc

answered Apr 06 '17 at 11:11

harsh tibrewal

785
6
21

Scripts entered as user data are executed as the root user. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html. – Santhosh Nagulanchi Nov 17 '18 at 02:53

score 1 · Answer 7 · answered Oct 30 '19 at 15:53

Adding this to the init script of the node will add environment variables on launch. They won't show up in the node configuration page but they will be able to use in any job.

#!/bin/bash
echo 'JAVA_HOME="/usr/lib/jvm/java-8-oracle/"' | sudo tee -a /etc/profile#

This answer is similar to what hamx0r proposed however, jenkins doesn't have permission to echo to /etc/profiles with or without sudo.

score 1 · Answer 8 · answered Apr 08 '20 at 15:30

This maynot be the exact answer to the OP's question but similar. I've thought of sharing this as I've wasted enough time searching for the answer and finally figured it out.

Example assuming - AWS EC2 running ubuntu.

If there is a scenario where you need to define the environment variables as well use it in the same bash session (same user-data process), then either you can add the variables to /etc/profile, /etc/environment or /home/ubuntu/.zshrc file. I have not tried /home/ubuntu/.profile file BTW.

Assuming adding to .zshrc file,

sudo su ubuntu -c "$(cat << EOF 
    echo 'export PATH="/tmp:\$PATH"' >> /home/ubuntu/.zshrc
    echo 'export ABC="XYZ"' >> /home/ubuntu/.zshrc
    echo 'export PY_VERSION=3.8.1' >> /home/ubuntu/.zshrc
    source /home/ubuntu/.zshrc
    echo printenv > /tmp/envvars  # To test
EOF
)"

Once the user data is finished running, you can see the environment variables which you have added in the script are echoed to the envvars file. Reloading the bash with source /home/ubuntu/.zshrc made the newly added variables available in the bash session.

(additional info) How to install zsh and oh-my-zsh?

sudo apt-get install -y zsh
sudo su ubuntu -c "$(cat << EOF 
    ssh-keyscan -H github.com >> /home/ubuntu/.ssh/known_hosts
    git clone https://github.com/robbyrussell/oh-my-zsh.git /home/ubuntu/.oh-my-zsh
    cp /home/ubuntu/.oh-my-zsh/templates/zshrc.zsh-template /home/ubuntu/.zshrc
    echo DISABLE_AUTO_UPDATE="true" >> /home/ubuntu/.zshrc
    cp /home/ubuntu/.oh-my-zsh/themes/robbyrussell.zsh-theme /home/ubuntu/.oh-my-zsh/custom
EOF
)"
sudo chsh -s /bin/zsh ubuntu

Wondering why I didn't added the environment variable in .bashrc? The scenario which I mentioned above (using the environment variables in the same user-data session) adding to .bashrc won't work. .bashrc is only sourced for interactive Bash shells so there should be no need for .bashrc to check if it is running in an interactive shell. So just like above,

source /home/ubuntu/.bashrc

won't reload the bash. You can check this out written right in the beginning of the .bashrc file,

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

score 1 · Answer 9 · answered Nov 25 '20 at 20:53

From this Medium.com article, you can put a script in UserData that writes to a file in /etc/profile.d that will get run automatically when a new shell is run.

Here is an example cloudformation.yaml

Parameters:
  SomeOtherResourceData:
    default: Fn::ImportValue: !Sub "SomeExportName"
Resources:
  WebApi:
    Type: AWS::EC2::Instance
    Properties:
      # ...
      UserData:
        Fn::Base64: !Sub
          - |
            #!/bin/bash

            cat > /etc/profile.d/load_env.sh << 'EOF'

            export ACCOUNT_ID=${AWS::AccountId}
            export REGION=${AWS::Region}
            export SOME_OTHER_RESOURCE_DATA=${SomeOtherResourceData}

            EOF
            chmod a+x /etc/profile.d/load_env.sh

And a YAML that exports something

# ...
Outputs:
  SomeExportName:
    Value: !Sub "${WebDb.Endpoint.Address}"
    Export:
      Name: SomeExportName

score 1 · Answer 10 · answered Jul 18 '22 at 13:07

1

Here is what working for me

   UserData:
    Fn::Base64:
      !Sub |
        #!/bin/bash
        echo "TEST=THISISTEST" >> /etc/environment

answered Jul 18 '22 at 13:07

Mehul Prajapati

61
1
5

score -4 · Answer 11 · edited Mar 12 '21 at 11:12

-4

The easiest way is definitely to use AWS Elastic Beanstalk, it creates for you everything you need with very small effort and have the easiest way in the entire AWS eco-system to set your environment variables.

check it out, there are also some exhaustive tutorials based on different languages https://docs.aws.amazon.com/elastic-beanstalk/index.html

edited Mar 12 '21 at 11:12

Dharman

30,962
25
85
135

answered Mar 12 '21 at 11:06

Filippo Secchi

51
1
4

This answer isn't related to the question. You could add suggestions in a comment but it's not an answer. – Deoxyseia Sep 09 '21 at 23:40

How to set Environment Variables on EC2 instance via User Data

11 Answers11

Linked