Copying large amount of files with nodejs?

Question

I have to copy a large amount of files (like 25000) asynchronously. I'm using this library: https://github.com/stephenmathieson/node-cp.

This is my code:

         for(var i = 0; i < 25000; i++){
            cp(origin[i], dest[i], function(err){
              console.log("successfully copied")
            })
         }

It completes the loop but it doens't copy every item. The "Successfully copied" is called between 6000 and 8000 times. After that it doesn't copy anymore. It has something to do with the memory or a limit for async tasks?

Any help would be appreciated!

score 2 · Answer 1 · answered Jun 21 '16 at 21:00

2

The copy function takes a callback which is usually a good clue that it's asynchronous. That means that the for loop will continue to run even though the copy hasn't completed, meaning you just queued up 25,000 copy operations!

There's a few ways to solve this but among the most common is using the async module.

var async = require('async');
async.forEachOf(origin, function (file, i, callback) {
    cp(file, dest[i], function(err){
        callback();
    });
})

This won't proceed to the next iteration of the loop until callback is called.

answered Jun 21 '16 at 21:00

Jack Guy

8,346
8
55
86

Thank you, but I also tried 'async'. Now I tried your code: `console.log(orig.length + " " + dst.length); async.forEachOf(orig, function (file, i, callback) { cp(file, dst[i], function(err){ console.log("finished:") callback(); }); })` orig.length and dest.length print 20000, but `finished` is printed between 7000 and 9000 times again. After that it does nothing :( – Oscar J. Irun Jun 21 '16 at 21:18
@OscarJ.Irun Hmm it seems that the module you're using for copy operations isn't very popular or well-maintained. Perhaps try one of the methods listed in this thread? http://stackoverflow.com/questions/11293857/fastest-way-to-copy-file-in-node-js – Jack Guy Jun 21 '16 at 21:34
I tried another too https://github.com/sindresorhus/cp-file, but this time it stops after copying between 4000 and 5000 files. Also tried the link you provided, but it doesnt copy half of the files neither :(. – Oscar J. Irun Jun 22 '16 at 00:44
@OscarJ.Irun are you sure you aren't getting errors when you copy some of the files? – Jack Guy Jun 22 '16 at 18:45
Pretty sure @Harangue Also when I make it synchronous, every file is copied successfully. – Oscar J. Irun Jun 23 '16 at 17:25

score 0 · Answer 2 · answered Jun 22 '16 at 03:13

you can copy async with this,

var fs = require('fs-extra')

fs.copy('/tmp/myfile', '/tmp/mynewfile', function (err) {
  if (err) return console.error(err)
  console.log("success!")
}) // copies file 

fs.copy('/tmp/mydir', '/tmp/mynewdir', function (err) {
  if (err) return console.error(err)
  console.log('success!')
}) // copies directory, even if it has subdirectories or file

info => https://www.npmjs.com/package/fs-extra

Copying large amount of files with nodejs?

2 Answers2