Can't download file from S3 in AWS Lambda without any HTTP error

Question

This is my code in AWS lambda:

import boto3
def worker_handler(event, context):

s3 = boto3.resource('s3')
s3.meta.client.download_file('s3-bucket-with-script','scripts/HelloWorld.sh', '/tmp/hw.sh')
print "Connecting to "

I just want to download a file stored in S3, but when I start the code, the program just run until timeout and print nothing on. This is the Logs file

START RequestId: 8b9b86dd-4d40-11e6-b6c4-afcc5006f010 Version: $LATEST
END RequestId: 8b9b86dd-4d40-11e6-b6c4-afcc5006f010
REPORT RequestId: 8b9b86dd-4d40-11e6-b6c4-afcc5006f010  Duration: 300000.12 ms  Billed Duration: 300000 ms  Memory Size: 128 MB Max Memory Used: 28 MB  
2016-07-18T23:42:10.273Z 8b9b86dd-4d40-11e6-b6c4-afcc5006f010 Task timed out after 300.00 seconds

I have this role in the this Lambda function, it shows that I have the permission to get file from S3
{
"Version": "2012-10-17",
"Statement": [
    {
        "Effect": "Allow",
        "Action": [
            "ec2:CreateNetworkInterface",
            "ec2:DescribeNetworkInterfaces",
            "ec2:DeleteNetworkInterface"
        ],
        "Resource": "*"
    },
    {
        "Effect": "Allow",
        "Action": [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents"
        ],
        "Resource": "arn:aws:logs:*:*:*"
    },
    {
        "Sid": "AllowPublicRead",
        "Effect": "Allow",
        "Action": [
            "s3:GetObject"
        ],
        "Resource": [
            "arn:aws:s3:::*"
        ]
    }
]
}

Is there any other set up I missed? Or anyway I can continue this program?

Answer 1

Per the docs your code should be a little different:

    import boto3

    # Get the service client
    s3 = boto3.client('s3')

    # Download object at bucket-name with key-name to tmp.txt
    s3.download_file("bucket-name", "key-name", "tmp.txt")

Also, note that Lambda has a ephemeral file structure, meaning downloading the file, does nothing really. You just downloaded it and then the Lambda shut down and ceased to exist, you need to send it somewhere after you download it to Lambda if you want to keep it.

Also, you may need to tweak your timeout settings to be higher.

Answer 2

As you mentioned a timeout, I would check the network configuration. If you are going through a VPC, this may be caused by the lack of route to the internet. This can be solved using a NAT Gateway or S3 VPC endpoint. The video below explains the configuration required.

Introducing VPC Support for AWS Lambda

Answer 3

As indicated in another answer you may need a NAT Gateway or a S3 VPC endpoint. I needed it because my Lambda was in a VPC so it could access RDS. I started going through the trouble of setting up a NAT Gateway until I realized that a NAT Gateway is currently $0.045 per hour, or about $1 ($1.08) per day, which is way more than I wanted to spend.

Then I needed to consider a S3 VPC endpoint. This sounded like setting up another VPC but it is not a VPC, it is a VPC endpoint. If you go into the VPC section there is a "endpoint" section (on the left) along with subnets, routes, NAT gateways, etc. For all the complexity (in my opinion) of setting up the NAT gateway, the endpoint was surprisingly simple.

The only tricky part was selecting the service. You'll notice the service names are tied to the region you are in. For example, mine is "com.amazonaws.us-east-2.s3"

But then you may notice you have two options, a "gateway" and an "interface". On Reddit someone claimed that they charge for interfaces but not gateways, so I went with gateway and things seem to work.

https://www.reddit.com/r/aws/comments/a6yppu/eli5_what_is_the_difference_between_interface/

If you don't trust that Reddit user, I later found that AWS currently says this: "Note: To avoid the NAT Gateway Data Processing charge in this example, you could setup a Gateway Type VPC endpoint and route the traffic to/from S3 through the VPC endpoint instead of going through the NAT Gateway. There is no data processing or hourly charges for using Gateway Type VPC endpoints. For details on how to use VPC endpoints, please visit VPC Endpoints Documentation."

https://aws.amazon.com/vpc/pricing/

Note, I also updated the pathing type per an answer in this other question, but I'm not sure that really mattered.

https://stackoverflow.com/a/44478894/764365

Answer 4

did you check if your time out was set correctly? I had the same issue, and it was timing out since my default value was set to 3 seconds and the file would take longer than that to download.

here is where you set your timeout setting：