how to pipe an archive (zip) to an S3 bucket

Question

I’m a bit confused with how to proceed. I am using Archive ( node js module) as a means to write data to a zip file. Currently, I have my code working when I write to a file (local storage).

var fs = require('fs');
var archiver = require('archiver');

var output = fs.createWriteStream(__dirname + '/example.zip');
var archive = archiver('zip', {
     zlib: { level: 9 }  
});

archive.pipe(output);
archive.append(mybuffer, {name: ‘msg001.txt’});

I’d like to modify the code so that the archive target file is an AWS S3 bucket. Looking at the code examples, I can specify the bucket name and key (and body) when I create the bucket object as in:

var s3 = new AWS.S3();
var params = {Bucket: 'myBucket', Key: 'myMsgArchive.zip' Body: myStream};
s3.upload( params, function(err,data){
    … 
});

Or 

s3 = new AWS.S3({ parms: {Bucket: ‘myBucket’ Key: ‘myMsgArchive.zip’}});
s3.upload( {Body: myStream})
    .send(function(err,data) {
    …
    });

With regards to my S3 example(s), myStream appears to be a readable stream and I am confused as how to make this work as archive.pipe requires a writeable stream. Is this something where we need to use a pass-through stream? I’ve found an example where someone created a pass-through stream but the example is too terse to gain proper understanding. The specific example I am referring to is:

Pipe a stream to s3.upload()

Any help someone can give me would greatly be appreciated. Thanks.

score 7 · Answer 1 · edited Feb 21 '19 at 19:52

This could be useful for anyone else wondering how to use pipe.

Since you correctly referenced the example using the pass-through stream, here's my working code:

1 - The routine itself, zipping files with node-archiver

exports.downloadFromS3AndZipToS3 = () => {
  // These are my input files I'm willing to read from S3 to ZIP them

  const files = [
    `${s3Folder}/myFile.pdf`,
    `${s3Folder}/anotherFile.xml`
  ]

  // Just in case you like to rename them as they have a different name in the final ZIP

  const fileNames = [
    'finalPDFName.pdf',
    'finalXMLName.xml'
  ]

  // Use promises to get them all

  const promises = []

  files.map((file) => {
    promises.push(s3client.getObject({
      Bucket: yourBubucket,
      Key: file
    }).promise())
  })

  // Define the ZIP target archive

  let archive = archiver('zip', {
    zlib: { level: 9 } // Sets the compression level.
  })

  // Pipe!

  archive.pipe(uploadFromStream(s3client, 'someDestinationFolderPathOnS3', 'zipFileName.zip'))

  archive.on('warning', function(err) {
    if (err.code === 'ENOENT') {
      // log warning
    } else {
      // throw error
      throw err;
    }
  })

  // Good practice to catch this error explicitly
  archive.on('error', function(err) {
    throw err;
  })

  // The actual archive is populated here 

  return Promise
    .all(promises)
    .then((data) => {
      data.map((thisFile, index) => {
        archive.append(thisFile.Body, { name: fileNames[index] })
      })

      archive.finalize()
    })
  }

2 - The helper method

const uploadFromStream = (s3client) => {
  const pass = new stream.PassThrough()

  const s3params = {
    Bucket: yourBucket,
    Key: `${someFolder}/${aFilename}`,
    Body: pass,
    ContentType: 'application/zip'
  }

  s3client.upload(s3params, (err, data) => {
    if (err)
      console.log(err)

    if (data)
      console.log('Success')
  })

  return pass
}

score 6 · Answer 2 · answered Aug 31 '20 at 12:13

The following example takes the accepted answer and makes it work with local files as requested.

const archiver = require("archiver")
const fs = require("fs")
const AWS = require("aws-sdk")
const s3 = new AWS.S3()
const stream = require("stream")

const zipAndUpload = async () => {
  const files = [`test1.txt`, `test2.txt`]
  const fileNames = [`test1target.txt`, `test2target.txt`]
  const archive = archiver("zip", {
    zlib: { level: 9 } // Sets the compression level.
  })
  files.map((thisFile, index) => {
    archive.append(fs.createReadStream(thisFile), { name: fileNames[index] })
  })
  const uploadStream = new stream.PassThrough()
  archive.pipe(uploadStream)
  archive.finalize()
  archive.on("warning", function (err) {
    if (err.code === "ENOENT") {
      console.log(err)
    } else {
      throw err
    }
  })
  archive.on("error", function (err) {
    throw err
  })
  archive.on("end", function () {
    console.log("archive end")
  })
  await uploadFromStream(uploadStream)
  console.log("all done")
}

const uploadFromStream = async pass => {
  const s3params = {
    Bucket: "bucket-name",
    Key: `streamtest.zip`,
    Body: pass,
    ContentType: "application/zip"
  }
  return s3.upload(s3params).promise()
}

zipAndUpload()

how to pipe an archive (zip) to an S3 bucket

2 Answers2

Linked