Append a file in S3 using Lambda

Question

How can I create and append a file in S3 using a lambda function? Also I want to limit the file size to 5 MB, is it possible?

Say, my lambda function searches for file, I want to keep a backup of the file searched. So I want to append the file whenever something is searched. If the file size increases above 5 MB I want to create a new file, which can be named according to the date and then start appending into it. How can I do that? I am using python/boto3. –

Does this answer your question? [AWS Lambda function write to S3](https://stackoverflow.com/questions/40188287/aws-lambda-function-write-to-s3) — st.huber, Jan 19 '22 at 08:35
Please Edit your Question to tell us more about your situation. For example, _WHY_ do you want to "append a file"? If you want to limit the file size, what would you do with the extra data above 5MB? What is the Lambda function doing? — John Rotenstein, Jan 19 '22 at 11:07
Say, my lambda function searches for file, I want to keep a backup of the file searched. So I want to append the file whenever something is searched. If the file size increases above 5 MB I want to create a new file, which can be named according to the date and then start appending into it. How can I do that? I am using python/boto3. — D S, Jan 21 '22 at 09:09

score 5 · Answer 1 · answered Jan 19 '22 at 08:37

Well you can not append any file in S3 as it is an "object" based storage. If you have to modify a file, you have to read it, append it, and save it again with same filename to update it on S3. There are tons of blogs on how to save file in s3 from lambda. It depends on your technology and framework.

One of such blogs: https://foobar123.com/serverless-save-a-file-in-s3-using-aws-lamdba-d3f087880dd2

score 1 · Answer 2 · answered Jan 19 '22 at 08:38

1

You can use the AWS SDK in Lambda functions to create a file in S3. For example, for Python, there is boto3.

However, I don't think there is a way to edit an S3 object directly.

You can download the file, append, and re-upload. Although this is not ideal because you keep downloading and re-uploading. Depending on the use case, you may want to consider other options that do not involve appending the file. For example, just create new objects with the additional lines.

answered Jan 19 '22 at 08:38

Register Sole

3,206
1
14
22

How can I create a new file using boto3 in a pre-existing bucket? – D S Jan 21 '22 at 09:00
There is an example for upload file in boto3 documentation https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html. – Register Sole Jan 21 '22 at 09:05
I am aware of that, but i wanted to know if I can create a file rather than uploading it? – D S Jan 21 '22 at 09:24
Sure. `client.put_object(Body=b"Let's write a file", Bucket="my_bucket_name", Key="hello.log")`. – Register Sole Jan 21 '22 at 09:28

score 1 · Answer 3 · answered Jan 19 '22 at 08:39

I don't recommand use lambda to upload file in S3 ( payload limit, time of treatment...)

I suggest you generate a pre-signed url from the lambda to upload the file directly to S3, the benefit of this is you upload the file directly from your local to S3 ( you can do a multipart upload, Transfert acceleration...).

The majority of projects that use lambdas for upload file, have faced this limitation.

Think of lambda like a rapid service to give you the url to upload, and you got another lambda that perform treatment once the file has been uploaded ( trigger from object put notification...)

Append a file in S3 using Lambda

3 Answers3