How to hash a download stream in Node js

Question

I want to know how to hash the download stream of a file using node js

Because I wanna hash the file before I store in to mongo db in order to avoid duplicates , I am using mongo grid fs by the way. https://github.com/aheckmann/gridfs-stream

downloading file

var download = function (url, dest, callback) {

                request.get(url)
                    .on('error', function (err) { console.log(err) })
                    .pipe(fs.createWriteStream(dest))
                    .on('close', callback);

            };

            final_list.forEach(function (str) {
                var filename = str.split('/').pop();

                console.log('Downloading ' + filename);

                download(str, filename, function () { console.log('Finished Downloading' + "" + filename) });
            });

`I want to hash the download stream of a file using node js` great, what's the question? — StudioTime, Oct 14 '19 at 05:49
you'll need to hash the file once it's finished writing -as was explained the [last time you asked the question](https://stackoverflow.com/questions/58333314/how-to-get-a-hash-or-checksum-of-a-file-from-a-stream-in-node-js-or-javascript) - and [the time before that](https://stackoverflow.com/questions/58319517/is-it-possible-to-get-the-md5-hash-value-of-a-file-being-downloaded-from-a-serve) — Bravo, Oct 14 '19 at 05:54
@Bravo , so i really need to create the file to my server right ? — Jhon Caylog, Oct 14 '19 at 06:28
no, you can store it in memory ... then write it if you need to ... you just can't pipe it directly to the db — Bravo, Oct 14 '19 at 07:19
@Bravo sir i just wanna ask how do we store it on the memory ? if you dont mind — Jhon Caylog, Oct 14 '19 at 07:20
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/200801/discussion-between-jhon-caylog-and-bravo). — Jhon Caylog, Oct 14 '19 at 07:44

score 1 · Accepted Answer · answered Oct 14 '19 at 06:01

1

function getHash(dest, filename) {
  let crypto = require('crypto');
  let hash = crypto.createHash('sha256').setEncoding('hex');
  let fileHash = "";
  let filePath = `${dest}/${filename}`
  fs.createReadStream(filePath)
    .pipe(hash)
    .on('finish', function() {
      fileHash = hash.read();
      console.log(`Filehash calculated for ${filename} is ${fileHash}.`);
      // insert into mongo db here
    });
}

answered Oct 14 '19 at 06:01

Vikash_Singh

1,856
2
14
27

does the files need to created or written into my server or disk ? – Jhon Caylog Oct 14 '19 at 06:09
cant we directly pipe it ? – Jhon Caylog Oct 14 '19 at 06:11
cause in this implementation i need to create the file first so it would be redundant that i store it in my server and then store it again in my databse – Jhon Caylog Oct 14 '19 at 06:39
yes you have to create file. other way you can't do that. – Vikash_Singh Oct 14 '19 at 07:36
so that file really must be created into my folder on my server ? – Jhon Caylog Oct 14 '19 at 07:37
other says that the file can be stored on a memory – Jhon Caylog Oct 14 '19 at 07:37
yes. you have to store file in you server. it can be in-memory or permanent storage. – Vikash_Singh Oct 14 '19 at 07:37
Sir mind if i ask if you can we store the file on the memory without creating it as file on the file system of my server – Jhon Caylog Oct 14 '19 at 07:43
cause it my current implementation now the file is being created in the file system before i can hash\ – Jhon Caylog Oct 14 '19 at 07:46
I wanted to store the file in memory and apply your example hash code – Jhon Caylog Oct 14 '19 at 07:50

How to hash a download stream in Node js

downloading file

1 Answers1