Serving zip file from s3 in aws lamba node.js - unable to extract after download

Question

I'm trying to create a lambda function to read a zip file from s3 and to serve it. But after downloading this file in the browser I can't unzip it, getting the error "Unable to extract, it is in an unsupported format". What can be a problem?

const file = await s3.getObject({ 
        Bucket: 'mybucket', 
        Key: `file.zip` 
    }).promise();

return {
    statusCode: 200,
    isBase64Encoded: true,
    body: Buffer.from(file.Body).toString('base64'),
    headers: {
        'Content-Type': 'application/zip',
        'Content-Disposition': `attachment; filename="file.zip"`,
    },
}

have you checked for the apigw settings enable binary content handling. if you are using this to download the zip folder, yes and @kelvin-schoofs mentioned change the format of buffer to base64 — sb39, Jul 29 '21 at 13:23

score 1 · Answer 1 · answered Jul 27 '21 at 16:26

Your file.Body should already be a Buffer, so Buffer.from(file.Body) should be unnecessary but unharmful.

I think your problem is that you're doing toString('base64') there. The documentation says:

If body is a binary blob, you can encode it as a Base64-encoded string by setting isBase64Encoded to true and configuring / as a Binary Media Type.

This makes me believe that it actually means that AWS will automatically convert your (non-base64) body into base64 in the response body. If that's the case, due to you doing .toString('base64'), your body is being base64'd twice. You could un-base64 your resulting file.zip and see what it gives.

score 0 · Accepted Answer · answered Jul 29 '21 at 20:31

0

The solution for me was to set 'Content-Encoding': 'base64' response header.

answered Jul 29 '21 at 20:31

mikach

2,427
2
14
15

1

That would mean AWS is still returning your body in base64, you're just telling your browser to decode it. That supports my theory that your `.toString('base64')` is the problem. As [this question and answers](https://stackoverflow.com/q/13265902/14274597) already mention, base64 isn't even a valid `Content-Encoding` option, so not every browser or http-client might understand it. _And it probably stops AWS from optimizing with `gzip`/`deflate`, if it automatically supports that._ – Kelvin Schoofs Jul 30 '21 at 12:46

score 0 · Answer 3 · answered Aug 03 '21 at 05:34

you can follow this code below

"use strict";

const AWS = require("aws-sdk");
const awsOptions = {
  region: "us-east-1",
  httpOptions: {
    timeout: 300000 // Matching Lambda function timeout
  }
};
const s3 = new AWS.S3(awsOptions);
const archiver = require("archiver");
const stream = require("stream");
const request = require("request");

const streamTo = (bucket, key) => {
  var passthrough = new stream.PassThrough();
  s3.upload(
    {
      Bucket: bucket,
      Key: key,
      Body: passthrough,
      ContentType: "application/zip",
      ServerSideEncryption: "AES256"
    },
    (err, data) => {
      if (err) throw err;
    }
  );
  return passthrough;
};

// Kudos to this person on GitHub for this getStream solution
// https://github.com/aws/aws-sdk-js/issues/2087#issuecomment-474722151
const getStream = (bucket, key) => {
  let streamCreated = false;
  const passThroughStream = new stream.PassThrough();

  passThroughStream.on("newListener", event => {
    if (!streamCreated && event == "data") {
      const s3Stream = s3
        .getObject({ Bucket: bucket, Key: key })
        .createReadStream();
      s3Stream
        .on("error", err => passThroughStream.emit("error", err))
        .pipe(passThroughStream);

      streamCreated = true;
    }
  });

  return passThroughStream;
};

exports.handler = async (event, context, callback) => {
  var bucket = event["bucket"];
  var destinationKey = event["destination_key"];
  var files = event["files"];

  await new Promise(async (resolve, reject) => {
    var zipStream = streamTo(bucket, destinationKey);
    zipStream.on("close", resolve);
    zipStream.on("end", resolve);
    zipStream.on("error", reject);

    var archive = archiver("zip");
    archive.on("error", err => {
      throw new Error(err);
    });
    archive.pipe(zipStream);

    for (const file of files) {
      if (file["type"] == "file") {
        archive.append(getStream(bucket, file["uri"]), {
          name: file["filename"]
        });
      } else if (file["type"] == "url") {
        archive.append(request(file["uri"]), { name: file["filename"] });
      }
    }
    archive.finalize();
  }).catch(err => {
    throw new Error(err);
  });

  callback(null, {
    statusCode: 200,
    body: { final_destination: destinationKey }
  });
};

score 0 · Answer 4 · answered Aug 03 '21 at 12:00

If you're not restricted to using the same URI as the URI that is serving your API, you could also create a pre-signed URL and return it as a redirection result. However, this will redirect to a different domain (S3 domain) so won't work out-of-the-box if you have to serve from the same domain name (e.g., because of firewall restrictions).

Serving zip file from s3 in aws lamba node.js - unable to extract after download

4 Answers4