1

I'm using Shell Command Activity that calls a Python script. This Python script utilizes boto3 to perform some functions. In the shell script in Shell Command Activity , I'm trying to install boto3 on to machine before calling my Python script. I'm also installing pip.

Inspite of installing boto3, I keep getting the error "ImportError: No module named boto3" , while it is executing my Python script when the data pipeline is activated This is how my shell script looks like:

#!/bin/bash
sudo yum -y install python37
curl -O https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py --user
pip install boto3 --upgrade --user
aws s3 cp s3://path-to-my-script/python_script.py /tmp/python_script.py
python /tmp/python_script.py

I also tried the recommendation mentioned here, which also failed with the same error: https://stackoverflow.com/a/44225052/4549186

(All of data pipeline activities run on the configured Ec2 resource that is created during pipeline activation)

What is the right way to install pip/boto3 on the ec2 resource and refer to it in the Python code?

N01
  • 39
  • 5
  • 2
    Seems you are mixing python and python3. Possibly pip and pip3. – Marcin Sep 03 '21 at 02:09
  • why use Data Pipeline in the first place? Seems like an unnecessary step in what could be a step function integrated with lambda https://docs.aws.amazon.com/step-functions/latest/dg/sample-lambda-orchestration.html If not an option then I would request additional logs from the ec2 instance about why it didn't work. Any error besides No module found? Can you make an instance with the same configuration and try the script manually? – jmp Sep 05 '21 at 13:51

0 Answers0