How do you upload, stream, and hash a file's contents in Node.js?

Question

I'd like to upload files on my server and name them according to their contents. This should be simple (it is in python), but I am having a hard time figuring out how to do it in Node.js.

I am using express and connect-form, which really just uses formidable. I also see that node has a library called crypto that is very similar to python's hashlib. Now I just need to understand how to stream the temp file connect-form gives me and hash it.

This is a Python/Flask(ish) implementation of what I'd like to do.

import hashlib
from Flask import request

def upload():
  file = request.files['file']
  hash = hashlib.sha256()
  name, ext = file.filename.rsplit('.', 1)
  try:
    for chunk in file.chunks()
      hash.update(chunk)
  finally:
    file.seek(0)
    new_name = "%s.%s" % (hash.hexdigest(),ext)
    file.save(os.path.join(UPLOAD_DIR, new_name))

I have seen a lot of these toy answers that just print out the file's name, but none that actually read and write the data.

https://gist.github.com/893110 at the bottom – Ilia Choly Oct 05 '11 at 05:19 — Ilia Choly, Oct 05 '11 at 05:19

score 3 · Accepted Answer · answered Oct 05 '11 at 07:57

Here it is in coffee-script in all its glory.

app.post '/upload', (request, response, next) ->
    request.form.complete (error, fields, files) ->
        if error
            next error
        else
            file = files.file
            [kind, extension] = file.type.split '/'
            hash = crypto.createHash 'sha256'
            stream = fs.createReadStream file.path,
                encoding:'binary'
            stream.addListener 'data', (chunk) ->
                hash.update chunk
            stream.addListener 'close', ->
                digest = hash.digest 'hex'
                new_filename = "#{digest}.#{extension}"
                new_path = "#{UPLOAD_DIR}/#{new_filename}"
                fs.rename file.path, new_path
                response.end new_filename

Rather than hacking the hash calculation into formidable, which would likely be more efficient but far more complicated, I opted to just re-read the file from its temporary location and hash that. Then instead of pumping it like in other examples, I just renamed the temp file into its new location.

score 1 · Answer 2 · answered Oct 05 '11 at 07:10

you can save a file like this:

var fs = require('fs'),
    util = require('util'),
    crypto = require('crypto');

// ...

req.form.complete(function (err, fields, files) {

    // ...

    var ext = files['content[media]']['filename'].split('.');
        ext = ext[ext.length-1];
        ext = ext.toLowerCase();

    var newFileName = req['connection']['remoteAddress'] + req['connection']['remotePort'] + Date.now();
    newFileName = crypto.createHash('md5').update(newFileName).digest("hex");
    newFileName += '.' + ext;

    var is = fs.createReadStream(files['content[media]']['path']);
    var os = fs.createWriteStream(app.set('dataDir') + '/' + newFileName);

    // copy file to public folder
    util.pump(is, os, function(error) {
        if (error) {
            console.log("Error copying file to public ... " + error);

            res.redirect("back");
            return;
        }
        else {
            // delete temp file
            fs.unlinkSync(files['content[media]']['path']);

            res.redirect('....');
        }
    });

});

Getting close, but that still doesn't name the file based on its contents. I think I've got this figured out though. — Nick Retallack, Oct 05 '11 at 07:45

How do you upload, stream, and hash a file's contents in Node.js?

2 Answers2