2

I use AWS Data Pipelines to run nightly SQL queries that populate tables for summary statistics. The UI's a bit funky, but eventually I got it up and working.

Now I'd like to do something similar with a python script. I have a file that I run every morning on my laptop (forecast_rev.py) but of course that means I have to turn on my laptop and kick this off every day. Surely I can schedule a Pipeline to do the same thing, and thus go away on vacation and not care.

For the life of me, I can't find a tutorial, AWS doc, or StackOverflow about this! I'm not even sure how to get started. Does anyone have a simple pipeline they'd be willing to share steps on?

ScottieB
  • 3,958
  • 6
  • 42
  • 60
  • 1
    Use [ShellCommandActivity](http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-shellcommandactivity.html) – franklinsijo Apr 17 '17 at 17:27

2 Answers2

2

I faced similar situation, here how i over come it.
I am going to describe how i did it with Ec2Resource. if you are looking for solution in EMRCluster refer @franklinsijo answer.

Steps
1. Store your python script in s3.
2. create a shell script(hello.sh)(given bellow) and store it to s3
3. Create a Ec2Resource Node and ShellCommandActivity Node and provide these information.

  • Provide shell script S3 url in "Script Uri" and set "stage" to true in ShellCommandActivity. And it should runs on your DefaultResource

Here is the shell script(hello.sh) which download your python program from s3 and stores locally, install python and required 3rd party library and finally execute your python file.

hello.sh

echo 'Download python file to local temp'
aws s3 cp s3://path/to/python_file/hello_world.py /tmp/hello.py
# Install python(on CentOs )
sudo yum -y install python-pip
pip install <dependencies>
python /tmp/hello.py

I had hard time while trying with bang line so do not included them here.
if aws cp command doesn't works(awscli is older), here is a quick solution for this case.

  1. Follow step 1-3 above, along with that create a s3DataNode.
    I. provide your python s3 url in "File Path" of S3DataNode.
    II. provide DataNode as "input" to ShellCommandActivity
    III. write following command in "command" field of ShellCommandActivity

Command

echo 'Install Python2'
sudo yum -y install python-pip
pip install <dependencies>
python ${INPUT1_STAGING_DIR}/hello_world.py
John
  • 1,212
  • 1
  • 16
  • 30
  • ERROR: Please define either: 'command' Or: 'scriptUri' on the component: 'ShellCommandActivityObj' – ScottieB Nov 16 '17 at 04:04
  • Please `Provide shell script S3 url in "Script Uri"`. please provide the details written after step 3. – John Nov 16 '17 at 10:30
  • Thanks for the idea, but I couldn't get it implemented. I ended up going a different way using a cron job on an ec2 instance. Not ideal, but it seems to hold. – ScottieB Nov 27 '17 at 05:47
  • @JanmejayaNanda I followed your instructions. When I activate the pipeline I keep getting 'status' skipped on 'type' ShellCommandActivity. As a test, I have python creating another table in database in my script, which is not being. Why I am getting this error – RustyShackleford Aug 10 '18 at 15:42
1
  1. You need to store your python script on S3 bucket
  2. Create Shell script that installs python and all your dependencies, copies your python script from S3 to local storage and runs it. Shell script example.
  3. Store this shell script on S3
  4. Use ShellCommandActivity to launch your shell script.

You can use this template as an example: http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-template-redshiftrdsfull.html It uses python script stored on s3 to convert MySQL schema to RedShift schema.

Example of python shell script that runs python program:

#!/bin/bash
curl -O https://s3.amazonaws.com/datapipeline-us-east-1/sample-scripts/mysql_to_redshift.py
python mysql_to_redshift.py
enisher
  • 199
  • 5