0

I have deployed an s3 bucket via terraform using the following code:

resource "aws_s3_bucket" "storage-dev" {
  bucket = "storage-dev"
}

resource "aws_s3_bucket_server_side_encryption_configuration" "storage-dev-encryption" {
  bucket = aws_s3_bucket.storage-dev.id

  rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
  }
}

resource "aws_s3_bucket_versioning" "storage-dev_versioning_policy" {
  bucket = aws_s3_bucket.storage-dev.id
  versioning_configuration {
    status = "Disabled"
  }
}

resource "aws_s3_bucket_public_access_block" "storage-dev_public_access" {
  bucket                  = aws_s3_bucket.storage-dev.id
  block_public_acls       = false
  block_public_policy     = false
  ignore_public_acls      = false
  restrict_public_buckets = false
}

output "s3_bucket_arn" {
  value       = aws_s3_bucket.storage-dev.arn
  description = "The ARN of the S3 bucket"
}

I have a serverless aurora postgres v1 database that I'm trying to interact with, and so far the easiest method I've found an example of is querying it from a lambda function (https://github.com/RekhuGopal/PythonHacks/blob/main/AWS_RDS_Arora_Serverless/Copy_s3_to_rds_arora_Serverless.py). I want to move some data from the s3 bucket to aurora, so I set up a lambda function in the same vpc as the aurora (an aws default vpc), and tried running the following code:

import json
import json
import os
import boto3
import csv

def lambda_handler(event, context):
    
    AWS_REGION = "##-####-1"
    client = boto3.client("s3", region_name=AWS_REGION)
    response = client.list_buckets()
    print("Listing Amazon S3 Buckets:")
    for bucket in response['Buckets']:
        print(f"-- {bucket['Name']}")
    print("donwloaded successfully....")
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Whenever I run the lambda it times out with the following error:

"errorMessage": "2023-05-26T11:37:58.483Z ffd29809-2238-####-9596-ef0c0fa07d37 Task timed out after 300.10 seconds"

As far as I know s3 buckets don't need to be in the same vpc as the resource trying to access them, so I can't figure out what's causing this connectivity issue. Any help would be much appreciated.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Boris
  • 716
  • 1
  • 4
  • 25
  • 2
    Amazon S3 buckets do not reside in a VPC. Instead, the S3 service is accessed either from the Internet or via a VPC Endpoint for S3. Thus, you either need a NAT Gateway to provide the Lambda function with Internet access, or you need to add a VPC Endpoint to that VPC. But, first I should ask... if your goal is to load data from S3 into the database, why not follow: [Importing data from Amazon S3 into an Aurora PostgreSQL DB cluster - Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_PostgreSQL.S3Import.html) – John Rotenstein May 26 '23 at 12:03
  • @John Rotenstein this is an aurora serverless v1 database, and so there is no http endpoint and no way to connect via psql or pgadmin as far as I know: https://stackoverflow.com/questions/62975793/why-cant-i-connect-to-a-new-aws-aurora-serverless-instance-from-my-pc – Boris May 26 '23 at 12:14
  • This question is a duplicate of https://stackoverflow.com/questions/39779962/access-aws-s3-from-lambda-within-vpc and https://stackoverflow.com/questions/54112422/access-aws-s3-from-lambda-within-default-vpc and https://stackoverflow.com/questions/60714724/how-lambda-connects-to-s3-inside-vpc and several others – Mark B May 26 '23 at 12:15
  • So is your question "How to import from S3 to an Aurora database", or is it "How to run a command on a database in a private subnet"? – John Rotenstein May 26 '23 at 23:07

0 Answers0