Execute python script at launch time on Amazon Linux 2

Question

I am trying to execute a python script on an Amazon Linux 2 instance. In my user-data section I have a script which copies the python script from an S3 bucket to the instance and executes it like so:

#!/bin/bash

# e - stops the script if there is an error
# x - output every command in /var/log/syslog
set -e -x

# set AWS region
echo "export AWS_DEFAULT_REGION=us-east-1" >> /etc/profile
source /etc/profile
# copy python script from the s3 bucket
aws s3 cp s3://${bucket_name}/ /home/ec2-user --recursive

sudo python3 my_python_script.py

The problem is that the python script doesn't seem to be getting executed at all.

Note: the python script gets copied fine from the bucket

What I am missing here?

UPDATE: after checking /var/log/cloud-init-output.log it looks like the problem is in the python script, it cannot find the boto3 module:

+ python3 /home/ec2-user/my_python_script.py
Traceback (most recent call last):
  File "/home/ec2-user/my_python_script.py", line 1, in <module>
    import boto3
ModuleNotFoundError: No module named 'boto3'
Dec 10 15:52:25 cloud-init[3697]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
Dec 10 15:52:25 cloud-init[3697]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Dec 10 15:52:25 cloud-init[3697]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed

The problem is that I do have boto3 module installed. I created a custom AMI image that does have all of the modules installed (I used pip3 to install them) before creating the custom AMI image

UPDATE2

I verified that the image does have boto3 package installed in the python3 library:

[ec2-user@ip-ip ~]$ python3
Python 3.7.9 (default, Aug 27 2020, 21:59:41)
[GCC 7.3.1 20180712 (Red Hat 7.3.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>>

UPDATE3

The cause of the problem was that I installed the boto3 package for my user only (i.e. pip3 install boto3 --user) and then I created the AMI image. So after adding the bellow line to my user-data script it worked fine

#!/bin/bash

...
sudo pip3 install boto3
sudo python3 my_python_script.py

Maybe log the output and stderr of that command to a file somewhere? What user is the userdata executing as, isn't it run as root already? — Tom Dalton, Dec 10 '20 at 15:35
I am pretty sure that when the instance launches it executes the user-data as root — Georgi Koemdzhiev, Dec 10 '20 at 15:37
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html "Scripts entered as user data are run as the root user, so do not use the sudo command in the script." — Tom Dalton, Dec 10 '20 at 15:37
Thanks, I will remove the `sudo` command. Howver, I did try without `sudo` before and the python script was still not getting executed :/ — Georgi Koemdzhiev, Dec 10 '20 at 15:39
Can you log the stdout and stderr output of the python3 invocation? — Tom Dalton, Dec 10 '20 at 15:40
Hmm. you're copying the bucket content to `/home/ec2-user` but I'm not sure if the userdata executes from that directory - so `python3 my_python_script.py` maybe just can't find `my_python_script.py` in the working directory? — Tom Dalton, Dec 10 '20 at 15:42
Thank you for your suggestion. I can try to see if I will be able to capture strerr from user-data (never done it before). As for, the script location, that's a good point. It may not be able to find it — Georgi Koemdzhiev, Dec 10 '20 at 15:45
@Asdfg thank you. I tried that but unfortunately, it is still not being called (I am using `htop` to verify if it runs as it should spike the memory usage quite a bit). I am looking at ways to capture any errors from `user-data` — Georgi Koemdzhiev, Dec 10 '20 at 15:53
check `/var/log/cloud-init-output.log` to see whats going on. Also try executing the script manually from `/var/lib/cloud/instances/<>` — Asdfg, Dec 10 '20 at 15:56
Thanks! I checked that file just now and it looks like the problem is in the python script itself :/ I will update the question — Georgi Koemdzhiev, Dec 10 '20 at 15:58
just do `pip install boto3` before you execute the python script. — Asdfg, Dec 10 '20 at 16:03
Thanks @Asdfg but that image do have that library installed. I did noticed that the `cloud-init` seems to be using `Python2.7` (I added more lines to the error output in my question). Do you know how I might force the user data to be using the already installed `Python3.8`? — Georgi Koemdzhiev, Dec 10 '20 at 16:07
‘python3’ uses Python 3.x version. You probably installed ’boto3’ with python2.7 when you created the image. — Asdfg, Dec 10 '20 at 16:10
I did verify that the image does have the `boto3` package installed already. I updated my question — Georgi Koemdzhiev, Dec 10 '20 at 16:14
I updated my question with the root cause of my issue. It was because I installed `boto3` package just for the `ec2-user` user and when the user-data script was getting executed, it was as `root` and it was not finding the `boto3` package — Georgi Koemdzhiev, Dec 10 '20 at 16:54

Chau Giang · Accepted Answer · 2020-12-10T15:56:48.493

1

You can redirect output to a file and read it to see the error: did you have python3, did your instance have credentianl/role to access this bucket, did you script requires any third party, can you try to run the script above as root in local first, the run command should be python3 /home/ec2-user/my_python_script.py?

edited Dec 10 '20 at 15:56

answered Dec 10 '20 at 15:35

Chau Giang

1,414
11
19

Thank you for your suggestion. However, I don't think that is the reason why my script is not getting executed. I have checked the `/etc/profile` file contents after the instance launches and it does contain my exported environment variable – Georgi Koemdzhiev Dec 10 '20 at 15:38
The instance does have credentials to copy the file from the bucket. I verified that after I sshed to the instance I saw the file in `home/ec2-user` – Georgi Koemdzhiev Dec 10 '20 at 15:54
Can you please send me the output of `cd ~ && ls` and `python3 /home/ec2-user/my_python_script.py` – Chau Giang Dec 10 '20 at 15:57
Thanks, I updated my question with the relevant data. It looks like my problem comes from my python script. It cannot find `boto3` (i.e. a library that my script relies on) – Georgi Koemdzhiev Dec 10 '20 at 16:02
1

you should install it first @GeorgiKoemdzhiev, I am not familiar with python maybe `pip install boto3` or `pip3 install boto3` before `sudo python3 /home/ec2-user/my_python_script.py` should help – Chau Giang Dec 10 '20 at 16:04
Thank you. I added a few more lines in my question and it looks like at boot time the version of python is `2.7` instead of the already installed `Python3.8` – Georgi Koemdzhiev Dec 10 '20 at 16:09
1

I think you can specific path to python3, for example `/usr/bin/python3 path/to/your/script` (I got this when use `which python3`) – Chau Giang Dec 10 '20 at 16:13
Thanks! I tried that and it did show that `python3` is located in a different place - `/usr/bin/python3`. I did try running that but it is still using `python2.7`. I checked `/var/log/cloud-init-output.log`, does that mean that I will just have to install `python3` as `root`? – Georgi Koemdzhiev Dec 10 '20 at 16:23
I updated my question with the root cause of my issue. It was because I installed `boto3` package just for the `ec2-user` user and when the user-data script was getting executed, it was as `root` and it was not finding the `boto3` package – Georgi Koemdzhiev Dec 10 '20 at 16:54
@GeorgiKoemdzhiev, it is my expected, I think you can add python3 to your PATH (when you build your image) in `~/.profile`, https://stackoverflow.com/a/14638025/10671200 – Chau Giang Dec 11 '20 at 02:04
Or you can use absolute path for python3 – Chau Giang Dec 11 '20 at 03:41
Thank you! Will try that :) – Georgi Koemdzhiev Dec 11 '20 at 08:03
1

You can try to add python3 to your PATH in /etc/environment also or use absolute python3 path instead of just python3 your/path/to/yourscript, just like other answer in the link above – Chau Giang Dec 11 '20 at 08:06
1

I will build a new image following your advice and see what happens :) Ideally, all of the packages should be available when I launch the server (i.e. I don't have to install them at boot time) – Georgi Koemdzhiev Dec 11 '20 at 08:09

score 0 · Answer 2 · answered Nov 27 '22 at 18:19

For anyone who is using cloudinit and terraform, this is how I managed it to work (one part of multipart cloudinit user data)-

part {
    content_type      = "text/x-shellscript"
    filename          = "run_src.sh"
    content           = <<-EOF
      #!/bin/bash
      cd /root
      mkdir tmp
      mkdir tmp/origin
      mkdir tmp/converted
      mkdir tmp/packaged
      
      pip3 install boto3
      cd src

      python3 main.py

    EOF
  }

And it works like charm.

Execute python script at launch time on Amazon Linux 2

2 Answers2