How can I download/upload multiple files at once using SFTP through the ssh2 Node.js module (within an Electron app)?

Question

I am building a simple SFTP client with Electron and I am attempting to download or upload multiple files at once using the ssh2 module and the SFTPStream within that module. I have tried many different method structures, some including use of es6-promise-pool. Every attempt I make results in one file from the array of files to transfer being transferred properly and then a subsequent

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 sftp_message listeners added to [EventEmitter]. Use emitter.setMaxListeners() to increase limit

message is displayed in the console and the rest of the files are not transferred. I am unsure how to change my method structure to prevent this from occurring. I am using ipcRenderer to tell ipcMain to execute the methods I will display here (here is my structure for uploading files for example).

let counter = 0;

// Upload local file to server
ipcMain.on('upload_local_files', (event, args) => { // args[0] is connection settings, args[1] is array of file paths
  let conn = new Client();
  conn.on('ready', () => {
    let pool = new PromisePool(uploadFileProducer(conn, args[1]), 10);
    pool.start().then(() => {
      conn.end();
      counter = 0;
      let tempArgs = [];
      tempArgs.push(curLocalDir);
      tempArgs.push(curRemoteDir);
      event.sender.send('local_upload_complete', tempArgs);
    });
  }).connect(args[0]);
});

// Producer used in promise pool to upload files
function uploadFileProducer(conn, files){
  if(counter < 100 && counter < files.length){
    counter++;
    return(uploadFile(conn, files[counter - 1]));
  }else{
    return null;
  }
}

// Function used to upload files in promise pool
function uploadFile(conn, file){
  return new Promise((resolve, reject) => {
    conn.sftp((error, sftp) => {
      return sftp.fastPut(file, curRemoteDir + file.substring(file.lastIndexOf('/') + 1), {}, (error) => {
        resolve(file);
      });
    });
  });
}

Admittedly, the use of promise pools is new to me and I am unsure if I am going about using them properly. Another post about this topic used promise pools to prevent the problem I am having from occurring, but that example did not involve an Electron app (I don't know if that's relevant). I appreciate any help I can get!

Are you sure nothing is uploaded? Because that's just a warning, and it's a normal one if you're uploading several files at once. — Marcos Casagrande, Dec 21 '19 at 00:46
Only the first file in the array is uploaded and that warning is given and then the connection is closed without uploading the rest of the files in the array — mandrews6975, Dec 21 '19 at 00:50
The warning has nothing to do with that, I'm posting an answer. — Marcos Casagrande, Dec 21 '19 at 00:51

Marcos Casagrande · Accepted Answer · 2019-12-21T01:07:32.527

The problem is not the Warning, which is just that, a warning, and normal in your current use case. The issue with the uploads is the incorrect usage of PromisePool.

I'm assuming you're using es6-promise-pool

You should pass a promise producer function to the constructor, but instead you're calling the function and passing a promise, that's why only a single files gets uploaded.

You should pass the producer without calling it, or make a producer that returns a function, or use a generator.

The PromisePool constructor takes a Promise-producing function as its first argument.

function *uploadFileProducer(conn, files) {
    for(const file of files)
        yield uploadFile(conn, file);
}

Now you can call:

let pool = new PromisePool(uploadFileProducer(conn, args[1]), 10)

And the PromisePool will iterate correctly the iterator returned by the generator function, and handle the concurrency accordingly.

You can also create a function that returns a Promise each call.

function uploadFileProducer(conn, files) {
  files = files.slice(); // don't want to mutate the original
  return () => uploadFile(conn, files.shift())
}

Regarding the warning, it's normal if you're uploading multiple things concurrently, if that's the case you can increase the limit using:

emitter.setMaxListeners(n)

How can I download/upload multiple files at once using SFTP through the ssh2 Node.js module (within an Electron app)?

1 Answers1