How to zip a directory, recursively, without getting malformed files when you try to unzip in node.js

Question

So far I have tried ADM-ZIP and easy-zip. Both of them create a zip that is mostly successful but has some malformed files:

This happens with a variety of file types (including just regular pictures and html pages). I suspect that the file type isn't even an issue. Either way, I can't use it if it doesn't work perfectly.

Some people have suggested node-archive but there are no instructions for how to zip a file, let alone recursively zip a directory while keeping its file structure.

update

as requested here is the code i am using (adm-zip)

var Zip = require("adm-zip");
var zip = new Zip();
zip.addLocalFolder("C:\\test");
zip.writeZip("C:\\test\\color.zip");

Answer 1

As you had mentioned, I would use node-archiver. You can get nested folders from node-archiver by:

var archiver = require('archiver');

var archive = archiver('zip')

// create your file writeStream - let's call it writeStream

archive.pipe(writeStream);

// get all the subfiles/folders for the given folder - this can be done with readdir
// take a look at http://nodejs.org/api/fs.html#fs_fs_readdir_path_callback
// and https://stackoverflow.com/questions/5827612/node-js-fs-readdir-recursive-directory-search
// appending to the archiver with a name that has slashes in it will treat the payload as a subfolder/file

// iterate through the files from the previous walk
ITERATE_THROUGH_RESULTS_FROM_WALK
    // look into http://nodejs.org/api/fs.html#fs_fs_readfile_filename_options_callback
    GET_READ_STREAM_ON_FILE_FROM_FS
        archive.append((new Buffer(data), 'utf-8'), {name: NAME_WITH_SLASHES_FOR_PATH});

archive.finalize(function (err) {
    // handle the error if there is one
});

Hopefully this is a pretty good nudge in the right direction (basically all of the steps are there for you). If it isn't clear yet:

1. Create your archive with the 'zip' option.
2. Pipe to a writestream that you created for where you want to save your zip file.
3. Walk through the folder recursively, saving paths in a collection (array) as you go along.
4. Iterate through this collection of paths, opening each file.
5. When you have the data of each file, create a Buffer from it and pass that as well as the path into archive.append.
6. Call archive.finalize.

For sake of readability, and because I believe I have given you pretty much all the steps you would need, I did not include all of the code (most notably the walk - which is laid out in the link anyhow).

Answer 2

You've asked for a zip but if you just need to archive files for transportation then I would suggest using tar.gz. Used it in production for transporting directories - works like a charm.

Here are usage examples: https://github.com/cranic/node-tar.gz#usage

var targz = require('tar.gz');
var compress = new targz().compress('/path/to/compress', '/path/to/store.tar.gz', function(err){
  if(err)
    console.log(err);
  console.log('The compression has ended!');
});

And uncompress:

var targz = require('tar.gz');
var compress = new targz().extract('/path/to/stored.tar.gz', '/path/to/extract', function(err){
  if(err)
    console.log(err);

  console.log('The extraction has ended!');
});

Answer 3

archiver has a bulk method which supports Grunt's "Files Array" format. Here's an example of recursively zipping folder foo and everything in it recursively.

foo
├── bar
|   ├── a.js
|   └── b.js
└── index.html

JavaScript based on this example in archiver

var output = fs.createWriteStream('foo.zip');
var archive = archiver('zip');

output.on('close', function() { console.log('done') });
archive.on('error', function(err) { throw err });

archive.pipe(output);

archive.bulk([
  { expand: true, cwd: 'foo', src: ['**/*'] }
]).finalize();

Answer 4

There is now an easier way to do it compared to the other answers.

archiver (page on npmjs) now has a method named 'directory' to help you do this.

Install archiver with:

npm install archiver --save

---

There are four things you need to do to recursively zip a directory and write it to a file.

A. Create an instance of archiver by using

    var ar = archive.create('zip',{});

B. Create a fs.writeStream and set up its event handlers...

var outputStream = fs.createWriteStream(odsFilepath, {flags: 'w'});
    outputStream.on('close', function () {
       console.log(ar.pointer() + ' total bytes written');
       console.log('ODS file written to:', odsFilepath);
    });
...

C. Wire up the output of our archiver to be piped to the writeStream we created:

ar.pipe(outputStream);

D. Ask our archiver to zip up the contents of the directory and place it in the 'root' or '/' of our zip file.

ar.directory(directoryPathToAddToZip, '/')
    .finalize();

---

Here is the code snippet of the function where I am using this. Note: put the following code snippet into a file, say index.js

var archiver = require('archiver');
var utility = require('utility');
var path = require('path');
var fs = require('fs');

//This is the directory where the zip file will be written into.
var outputDirname = ".";

//This directory should contain the stuff that we want to
//  put inside the zip file
var pathOfContentDirToInsertIntoArchive = "./tozip";

saveToZip(function () {
        console.log('We are now done');
});

function saveToZip(done) {
    //we use the utility package just to get a timestamp string that we put
    //into the output zip filename
    var ts = utility.logDate();
    var timestamp = ts.replace(/[ :]/g, '-');
    var zipFilepath = path.normalize(outputDirname + '/' + timestamp + '.zip')

    var ar = archiver.create('zip', {});

    var output = fs.createWriteStream(zipFilepath, {flags: 'w'});
    output.on('close', function () {
        //console.log(ar.pointer() + ' total bytes');
       console.log('ZIP file written to:', zipFilepath);
        return done(null, 'Finished writing')
    });

    ar.on('error', function (err) {
        console.error('error compressing: ', err);
        return done(err, 'Could not compress');
    });

    ar.pipe(output);
    ar.directory(path.normalize(pathOfContentDirToInsertIntoArchive + '/'), '/')
    .finalize();
}

Then do a npm install archiver --save and npm install utility --save

Then create a directory named "tozip" in the current directory and put in there some files that you want to compress into the output zip file.

Then, run node index.js and you will see output similar to:

$ node index.js
ZIP file written to: 2016-12-19-17-32-14.817.zip
We are now done

The zip file created will contain the contents of the tozip directory, compressed.