The environment variable AWS_ACCESS_KEY_ID must be set

Question

I am using Linux 18.04 and I want to lunch a spark cluster on EC2.

I used the export command to set environment variables

export AWS_ACCESS_KEY_ID=MyAccesskey
export AWS_SECRET_ACCESS_KEY=Mysecretkey

but when I run the command to lunch the spark cluster I get

ERROR: The environment variable AWS_ACCESS_KEY_ID must be set

I put all the commands I used in case I made a mistake:


sudo mv ~/Downloads/keypair.pem   /usr/local/spark/keypair.pem
sudo mv ~/Downloads/credentials.csv   /usr/local/spark/credentials.csv
# Make sure the .pem file is readable by the current user.
chmod 400 "keypair.pem"
# Go into the spark directory and set the environment variables with the credentials information
cd spark
export AWS_ACCESS_KEY_ID=ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=SECRET_KEY
# To install Spark 2.0 on the cluster:
sudo spark-ec2/spark-ec2 -k keypair --identity-file=keypair.pem --region=us-west-2 --zone=us-west-2a --copy-aws-credentials --instance-type t2.micro --worker-instances 1 launch project-launch

I am new to these things and any help is really appreciated

You've set the AWS environment variables in your current environment but then started a new environment using `sudo`. See https://stackoverflow.com/questions/8633461/how-to-keep-environment-variables-when-using-sudo — jarmod, Jan 05 '20 at 20:19
@marxmacher Yes I have that file You mean just writing source bashrc. ? — G. Hak., Jan 05 '20 at 21:31

score 4 · Answer 1 · answered Jun 05 '20 at 10:48

You can also retrieve the value AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY by using the get subcommand for aws configure:

AWS_ACCESS_KEY_ID=$(aws configure get aws_access_key_id)
AWS_SECRET_ACCESS_KEY=$(aws configure get aws_secret_access_key)

In command line:

sudo AWS_ACCESS_KEY_ID=$(aws configure get aws_access_key_id) AWS_SECRET_ACCESS_KEY=$(aws configure get aws_secret_access_key) spark-ec2/spark-ec2 -k keypair --identity-file=keypair.pem --region=us-west-2 --zone=us-west-2a --copy-aws-credentials --instance-type t2.micro --worker-instances 1 launch project-launch

source: AWS Command Line Interface User Guide

score 3 · Accepted Answer · answered Jan 05 '20 at 21:38

Environment variables can be simply passed after sudo in form ENV=VALUE and they'll be accepted by followed command. It's not known to me if there are restrictions to this usage, so my example problem can be solved with:

sudo AWS_ACCESS_KEY_ID=ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=SECRET_KEY spark-ec2/spark-ec2 -k keypair --identity-file=keypair.pem --region=us-west-2 --zone=us-west-2a --copy-aws-credentials --instance-type t2.micro --worker-instances 1 launch project-launch

The environment variable AWS_ACCESS_KEY_ID must be set

2 Answers2