How should I post a file to AWS Lambda function, process it, and return a file to the client?

Question

I'm using serverless-http to make an express endpoint on AWS Lambda - pretty simple in general. The flow is basically:

1. POST a zip file via a multipart form to my endpoint
2. Unzip the file (which contains a bunch of excel files)
3. Merge the files into a single Excel file
4. res.sendFile(file) the file back to the user

I'm not stuck on this flow 100%, but that's the gist of what I'm trying to do.

Lambda functions SHOULD give me access to /tmp for storage, so I've tried messing around with Multer to store files there and then read the contents, I've also tried the decompress-zip library and it seems like the files never "work". I've even tried just uploading an image and immediately sending it back. It sends back an files called incoming.[extension], but it's always corrupt. Am I missing something? Is there a better way to do this?

Answer 1

Typically when working with files the approach is to use S3 as the storage, and there are a few reasons for it, but one of the most important is the fact that Lambda has an event size limit of 6mb, so you can't easily POST a huge file directly to it.

If your zipped excel files is always going to be less than that, then you are safe on that regard. If not, then you should look into a different flow, maybe something using AWS step functions with Lambda and S3.

Concerning your issue with unzipping the file, I have personally used and can recommend adm-zip, which would look something like this:

//unzip and extract file entries
var zip = new AdmZip(rawZipData);
var zipEntries = zip.getEntries();
console.log("Zip contents : " + zipEntries.toString());
zipEntries.forEach(function(entry){
    var fileContent = entry.getData().toString("utf8");
});