2

I need to save files I get from S3 into a Lambda's file system and I wanted to know if I can do that simply using fs.writeFileSync ?

Or do I have to still use the context function as described here: How to Write and Read files to Lambda-AWS with Node.js

(tried to find newer examples, but could not).

What is the recommended method?

Please advise.

David Faizulaev
  • 4,651
  • 21
  • 74
  • 124
  • 1
    Not sure what you mean by the "context function". Yes, you can use the typical `fs` functions, but writing is limited to /tmp and the max diskspace available to your Lambda function in that location is 512 MB. Files written there may persist to the next (warm) Lambda invocation. – jarmod Oct 07 '21 at 16:34
  • If simply copying from S3 to local disk then stream the content ([example](https://stackoverflow.com/a/33938065/271415)). – jarmod Oct 07 '21 at 16:40
  • If I steam the content, what diskspace limit do I have? – David Faizulaev Oct 07 '21 at 17:38
  • You can write up to 512MB in /tmp. Depending on your app, you may be able to process the entire S3 object in RAM via streaming, without any need to persist to disk. – jarmod Oct 07 '21 at 17:43

2 Answers2

6

Yes, you can use the typical fs functions to read/write from local disk, but be aware that writing is limited to the /tmp directory and the default max diskspace available to your Lambda function in that location is 512 MB. Also note that files written there may persist to the next (warm) Lambda invocation.

If you want to simply download an object from S3 to the local disk (assuming it will fit in the available diskspace) then you can combine AWS SDK methods and Node.js streaming to stream the content to disk.

Also, it's worth noting that, depending on your app, you may be able to process the entire S3 object in RAM via streaming, without any need to actually persist to disk. This is helpful if your object size is over 512MB.

Update: as of March 2022, you can now configure Lambda functions with more ephemeral storage in /tmp, up to 10 GB. You get 512 MB included with your Lambda function invocation and are charged for the additional configured storage above 512 MB.

If you need to persist very large files, consider using Elastic File System.

jarmod
  • 71,565
  • 16
  • 115
  • 122
  • 1
    Note that as of March 2022 AWS allows you to adjust the amount of /tmp storage your lambda has access to up to 10GB https://aws.amazon.com/blogs/aws/aws-lambda-now-supports-up-to-10-gb-ephemeral-storage/ – adam8810 Sep 01 '22 at 21:57
  • Thanks @adam8810, have updated answer. – jarmod Sep 01 '22 at 23:28
1

Lambda does not allow access to the local file system. It is mean to be an ephemeral environment. They allow access to the /tmp folder, but only for a maximum of 512MB. If you want to have storage along with your function, you will need to implement AWS S3 or AWS EFS.

Here's an article from AWS explaining this.

Here's the docs on adding storage to Lambda.

zbauman
  • 217
  • 2
  • 8
  • 1
    AWS Lambda *does* allow access to the local file system. The effective (sandbox) user may not necessarily have sufficient permission to access the entire file system but it can certainly access all-readable files e.g. in /etc. – jarmod Oct 07 '21 at 17:41
  • 1
    @jarmod that’s a good distinction, thanks – zbauman Oct 07 '21 at 18:12
  • Not sure I understand, I can direct a lambda to use a S3 bucket as its storage? – David Faizulaev Oct 07 '21 at 18:31