20

I am using API gateway to call lambda function that imports a mpeg file (10 mb) from s3 and saves in /tmp folder of lambda and the lambda uploads it to external API (youtube etc) Recently the API gateway call to lambda is failing intermittently with error

[Errno 28] No space left on device

Here is how i am downloading the file

urllib.urlretrieve (s3_mpeg_url, '/tmp/{}'.format(mpeg_filename))

If i create a new version of that same lambda function and assign to alias API gateway pointing to , it starts to work and again at some point it keeps getting the same error

When i test that lambda function from lambda console it always works

Any idea ?

General Grievance
  • 4,555
  • 31
  • 31
  • 45
codeexplorer
  • 409
  • 1
  • 6
  • 19

2 Answers2

22

Your lambda function has approximately 500MB of disk space available on /tmp. However, for performance reasons, AWS Lambda might retain and reuse instances of your function on subsequent calls. There are two possible paths you can take here:

  1. If your function is totally stateless (i.e. you don't need the mpeg file after uploading it to the external API), just delete it from the /tmp folder after the upload.
  2. If you need to keep the data around, consider storing it on another media, like S3 or DynamoDB after processing.
Viccari
  • 9,029
  • 4
  • 43
  • 77
  • Thanks .. I am looking into option 1 you mentioned but it didnt helped...please pardon my lack of knowledge on deleting files on aws servers... i tried os.remove(/tmp/*.mpeg) but its not working(i am using python in lambda). Can you let me know how we can clear tmp folder – codeexplorer Jan 19 '18 at 19:01
  • Don't believe that os.remove() supports wildcards. Delete the file by name, or use some feature of glob. – jarmod Jan 19 '18 at 20:23
  • that may be it.it wont support wildcards.I used following to get it work `def emptyDir(folder): fileList = os.listdir(folder) for f in fileList: filePath = folder + '/'+f if os.path.isfile(filePath): os.remove(filePath) elif os.path.isdir(filePath): newFileList = os.listdir(filePath) for f1 in newFileList: insideFilePath = filePath + '/' + f1 if os.path.isfile(insideFilePath):\n os.remove(insideFilePath)` – codeexplorer Jan 19 '18 at 20:51
  • Right now, you can configure your ephemeral storage between **512MB** and **10,240MB**. Have in mind that the ephemeral storage is available in each function’s **/tmp** directory. – robe007 Jun 21 '22 at 19:44
2

You can increase the storage from 512MB (default) up to 10,240MB.

GUI: how to change your storage

CLI:

 aws lambda update-function-configuration --function-name <function_name> \
          --ephemeral-storage '{"Size": 10240}'

Note that you are charged for any additional size you incremented.

See full article here: https://aws.amazon.com/blogs/aws/aws-lambda-now-supports-up-to-10-gb-ephemeral-storage/

trex
  • 325
  • 3
  • 8