Write csv file and save it into S3 using AWS Lambda (python)

Question

I'm trying to write a csv file into an S3 bucket using AWS Lambda, and for this I used the following code:

data=[[1,2,3],[23,56,98]]
with open("s3://my_bucket/my_file.csv", "w") as f:
   f.write(data)

And this raises the following error:

[Errno 2] No such file or directory: u's3://my_bucket/my_file.csv': IOError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 51, in lambda_handler
with open("s3://my_bucket/my_file.csv", "w") as f:
IOError: [Errno 2] No such file or directory: u's3://my_bucket/my_file.csv'

Can I have some help with this please ?

PS: I'm using python 2.7

Thanking you in advance

Lambda doesn't have native device driver support for s3:// URIs like that. Write the CSV file to local file system (/tmp) and then use boto3's put_object() method. You can also stream the file contents into S3 using boto3, if preferred. — jarmod, Apr 06 '18 at 14:33
See https://stackoverflow.com/questions/40336918/how-to-write-a-file-or-data-to-an-s3-object-using-boto3 — jarmod, Apr 06 '18 at 14:47

score 14 · Answer 1 · answered Dec 24 '19 at 00:03

Better to answer later than never. There are four steps to get your data in S3:

Call the S3 bucket
Load the data into Lambda using the requests library (if you don't have it installed, you are gonna have to load it as a layer)
Write the data into the Lambda '/tmp' file
Upload the file into s3

Something like this:

import csv
import requests
#all other apropriate libs already be loaded in lambda

#properly call your s3 bucket
s3 = boto3.resource('s3')
bucket = s3.Bucket('your-bucket-name')
key = 'yourfilename.txt'

#you would need to grab the file from somewhere. Use this incomplete line below to get started:
with requests.Session() as s:
    getfile = s.get('yourfilelocation')

#Only then you can write the data into the '/tmp' folder.
with open('/tmp/yourfilename.txt', 'w', newline='') as f:
    w = csv.writer(f)
    w.writerows(filelist)
#upload the data into s3
bucket.upload_file('/tmp/yourfilename.txt', key)

Hope it helps.

For case with concurrent invocations of the lambda - won't it create collisions using same '/tmp/yourfilename.txt' ? — Alex_Y, Jun 28 '22 at 14:35
It certainly will but one option would be to limit this Lambda to only 1 execution at a time. — Mr Chow, Oct 01 '22 at 16:21

score -3 · Answer 2 · edited Aug 19 '19 at 12:58

I am not aware of using AWS Lambda, but I have been using Boto3 to do the same. It is a simple few line code.

#Your file path will be something like this:
#s3://<your_s3_bucket_name>/<Directory_name>/<File_name>.csv

import boto3

BUCKET_NAME = '<your_s3_bucket_name>'
PREFIX = '<Directory_name>/'
s3 = boto3.resource('s3')
obj = s3.Object(BUCKET_NAME, PREFIX + '<File_name>.csv')
obj.put(Body=content)

score -5 · Answer 3 · answered Apr 06 '18 at 14:30

-5

with open("s3://my_bucket/my_file.csv", "w+") as f:

instead of

with open("s3://my_bucket/my_file.csv", "w") as f:

notice the "w" has changed to "w+" this means that it will write to the file, and if it does not exist it will create it.

answered Apr 06 '18 at 14:30

Ryan

28
4

Okay so does the s3://my_bucket/ directory actually exist? – Ryan Apr 06 '18 at 15:19
Yesn but the file does not exist – kab Apr 06 '18 at 15:22
The `s3://my_bucket/` "directory" doesn't exist locally... It's on S3. And it's not even a directory, it's an S3 bucket. It can't be accessed like a local file, you have to use boto3. – Mark B Apr 06 '18 at 19:04

Write csv file and save it into S3 using AWS Lambda (python)

3 Answers3