15

I'm using client.upload in pkgcloud to upload a directory of files. How can I execute a callback after all the streams have finished? Is there a built-in way to register each stream's "finish" event and execute a callback after they have all fired?

var filesToUpload = fs.readdirSync("./local_path"); // will make this async

for(let file of filesToUpload) {
    var writeStream = client.upload({
        container: "mycontainer",
        remote: file
    });
    // seems like I should register finish events with something
    writeStream.on("finish", registerThisWithSomething);
    fs.createReadStream("./local_path/" + file).pipe(writeStream);
}
Avocado
  • 871
  • 6
  • 23
berg
  • 614
  • 3
  • 9
  • 23
  • You can use async.js for this kind of problem. You can use method provided by async.js as `async.waterfall()`.The result of one function will be passed to another as a callback arguments.You should check out the documentation of it . – Kartikeya Sharma May 14 '16 at 17:50

2 Answers2

30

One way to do it, is to generate a Promise task for each upload, then utilizing Promise.all().

Assuming you are using ES6, then the code would look something like this:

    const uploadTasks = filesToUpload.map((file) => new Promise((resolve, reject) => {
        var writeStream = client.upload({
            container: "mycontainer",
            remote: file,
        });
        // seems like i should register finish events with something
        writeStream.on("finish", resolve);
        fs.createReadStream("./local_path/" + file).pipe(writeStream);
    }));

    Promise.all(uploadTasks)
      .then(() => { console.log('All uploads completed.'); });

Alternatively, if you have access to async / await - you can utilize this. For example:

    const uploadFile = (file) => new Promise((resolve, reject) => {
      const writeStream = client.upload({
        container: "mycontainer",
        remote: file,
      });
      writeStream.on("finish", resolve);
      fs.createReadStream("./local_path/" + file).pipe(writeStream);
    });
    
    const uploadFiles = async (files) => {
      for(let file of files) {
        await uploadFile(file);
      }
    }

    await uploadFiles(filesToUpload);
    console.log('All uploads completed.');
JAM
  • 6,045
  • 1
  • 32
  • 48
  • For those who are interested, I had a slightly different problem while handling streams in hummus.js and this answer helped me solve my problem! Thanks @JAM my issue: https://stackoverflow.com/questions/51731191/how-to-modify-a-stream-from-request-to-a-new-response-stream-in-hummusjs – Jay Edwards Aug 08 '18 at 09:24
  • There is a closing parenthesis missing after the end of the promise (in both examples, which is kind of interesting), but SO won't let me add it because edits must comprise at least 6 characters. – sgrubsmyon May 08 '23 at 12:35
  • Good catch @sgrubsmyon - I've added the missing parenthesis – JAM May 10 '23 at 10:45
0

Take a look at NodeDir，which has methods like readFilesStream / promiseFiles etc.

B.Ma
  • 839
  • 7
  • 9