Run Python Script on AWS and transfer 5GB of files to EC2

Question

I am an absolute beginner in AWS: I have created a key and an instance, the python script I want to run in the EC2 environment needs to loop through around 80,000 filings, tokenize the sentences in them, and use these sentences for some unsupervised learning.

This might be a duplicate; but I can't find a way to copy these filings to the EC2 environment and run the python script in EC2, I am also not very sure as to how I can use boto3. I am using Mac OS. I am just looking for any way to speed things up. Thank you so so much! I am forever grateful!!!

Possible duplicates: https://stackoverflow.com/questions/10364950/uploading-files-on-amazon-ec2 https://stackoverflow.com/questions/6331756/how-to-transfer-files-to-amazon-ec2-instance-from-my-windows-7-computer — Mark B, Feb 10 '20 at 22:05
Where do these "filings" come from? Are they stored in Amazon S3? — John Rotenstein, Feb 11 '20 at 03:06

score 0 · Answer 1 · answered Feb 10 '20 at 22:07

Here's one way that might help:

create a simple IAM role that allows S3 access to the bucket holding your files
apply that IAM role to the running EC2 instance (or launch a new instance with the IAM role)
install the awscli on the EC2 instance
SSH to the instance and sync the S3 files to the EC2 instance using aws s3 sync
run your app

I'm assuming you've launched EC2 with enough diskspace to hold the files.

score 0 · Accepted Answer · edited Feb 11 '20 at 14:51

Here's what I tried recently:

Create the bucket and keep the bucket accessible for public.
Create the role and add HTTP option.
Upload all the files and make sure the files are public accessible.
Get the HTTP link of the S3 file.
Connect the instance through putty.
wget copies the file into EC2 instance.

If your files are in zip format, one time copy enough to move all the files into instance.

Run Python Script on AWS and transfer 5GB of files to EC2

2 Answers2