1

My aim is to launch an instance such that a start-up script is triggered on boot-up to download some configuration files stored in AWS S3. Therefore, in the start-up script, I am setting the S3 bucket details and then, triggering a config.sh where aws s3 sync does the actual download. However, the aws command does not work - it is not found for execution.

User data

I have the following user data when launching an EC2 instance:

#!/bin/bash
# Set command from https://stackoverflow.com/a/34206311/919480
set -e -x
export S3_PREFIX='bucket-name/folder-name'
/home/ubuntu/app/sh/config.sh

The AWS CLI was installed with pip as described in the documentation.

Observation

I think, the user data script is run with root user ID. That is why, in the user data I have /home/ubuntu/ because $HOME did not resolve into /home/ubuntu/. In fact, the first command in config.sh is mkdir /home/ubuntu/csv which creates a directory with owner as root!

So, would it be right to conclude that, the user data runs under root user ID?

Resolution

Should I use REST API to download?

cogitoergosum
  • 2,309
  • 4
  • 38
  • 62
  • What do you mean by "The AWS CLI was installed with pip"? It is not in your User Data script. Was it installed, then you created an AMI, then you used that AMI with this new instance? Or is it inside the `config.sh` file? You can always try `sudo su -` to become root and then paste your User Data to see if it runs. – John Rotenstein Jun 01 '18 at 04:39
  • @JohnRotenstein AWS CLI was installed on the AMI. – cogitoergosum Jun 01 '18 at 04:42

2 Answers2

4

Scripts entered as user data are executed as the root user, so do not use the sudo command in the script.

See: Running Commands on Your Linux Instance at Launch

One solution is to set the PATH env variable to include AWS CLI (and add any other required path) before executing AWS CLI.

helloV
  • 50,176
  • 7
  • 137
  • 145
  • `sudo` is not used any where. Will try `PATH` option. – cogitoergosum Jun 01 '18 at 04:29
  • Or, just refer to `aws` via the full path, eg `/usr/local/bin/aws s3 cp...` – John Rotenstein Jun 01 '18 at 04:55
  • @JohnRotenstein for me `which aws` shows `/home/ubuntu/.local/bin/aws` - probably because I didn't do a `sudo pip install`. Therefore, I tried with `su ubuntu -` right at the start of the script and now I see this, `__init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'b'su ubuntu -'...'` – cogitoergosum Jun 01 '18 at 07:59
  • Also, providing the full path `/home/ubuntu/.local/bin/aws` does not help because `import` of AWS CLI driver fails for `root`. Hence, the choice of using `su ubuntu -`. – cogitoergosum Jun 01 '18 at 08:23
1

Solution

Given that, AWS CLI was installed without a sudo pip, the CLI is not available for root. Therefore, to run with ubuntu user, I used the following user data script:

#!/bin/bash
su ubuntu -c '$HOME/app/sh/config.sh default`

In config.sh, the argument default is used to build the full S3 URI before invoking the CLI. However, the invocation was successful only with the full path $HOME/.local/bin/aws despite the fact that aws can be accessed with normal login.

cogitoergosum
  • 2,309
  • 4
  • 38
  • 62
  • Having the same problem -- `aws s3` not found in the user data environment. But I've come across this [passage](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html): " If you use an AWS API, including the AWS CLI, in a user data script, you must use an instance profile when launching the instance. An instance profile provides the appropriate AWS credentials required by the user data script to issue the API call." Could that be related? – Martynas Jusevičius Jun 10 '21 at 21:06