Chaining routes in NodeJs with values after sending API response

Question

I want to chain routes in NodeJs with values after sending API response to end-ser,

WHY: > The uploaded files would be somewhat large (5-50mb each) and require some processing, can not make my API user wait/timeout while my NodeJS code is working.. so need, 1: Upload files and send success immediately to user, Process files (few promises) and return/log success/failure for notification system.

My individual code blocks are done and working fine (i.e. upload service and file processing service both are good under tests and work nicely when tested individually.)

now with the API to upload in place, I've added following code:

router.post('/upload', upload.array('upload_data', multerMaxFiles), (req, res, next) => {
 ////some uploading and processing stuff - works nicely
 res.json({ 'message': 'File uploaded successfully.' });// shown to API client nicely
 console.log("what next? " + utilz.inspect(uploaded_file_paths)) //prints file names on console
 next();
});

PROBLEM:

app.use('/api', uploadRoute); //The above code route

//want some processing to be done

app.use(function(req, res, next) {

**want those uploaded file names here**
tried with few response object options but stabs with error
});

OR use something like ....

app.use(someFunction(uploaded_file_names)); **want those uploaded file names as params**

PS: Any promise after the file upload success would result in 'Error: Can't set headers after they are sent.', so not helpful writing anything there.

Any suggestions folks.

-- N Baua

Answer 1

Once you've sent a response back to the browser (to keep it from timing out during your long processing time), that http request is done. You cannot send any more data on it and trying to do so will trigger a server-side error. You cannot "chain routes" the way you were asking as you seem to want to do because you simply can't send more data over that http request once you've sent the first response.

There are two common ways to deal with this issue.

1. As part of your initial response, send back a transaction ID and then have the client poll back every few seconds with an Ajax call asking what the final status is of that transaction. The server can return "in progress" until it is finally done and then it can return the final status.

2. You can connect a webSocket or socket.io connection from client to server. As part of your initial response to the upload, send back a transaction ID. Then, when the transaction is done server-side, it sends a notification on the webSocket or socket.io connection for that particular client with the transactionID with the final status. The client can then respond accordingly to that final status. You can either keep the webSocket/socket.io connection open for use with other requests or you can then close that connection.

Using either technique, you could also return/send a progress value (like percent complete) that the client could use to display completion progress. This is generally very helpful on the client-side to keep an impatient user from giving up or refreshing the page. If they can see that the processing is proceeding, they won't give up thinking that maybe it stopped working.

Answer 2

This should work with res.write(). But it does depend on your clients cache i think.

I tried this, but it does not work in my firefox.

app.get('/test', function(req, res) {
  var count     = 0;
  var interval  = setInterval(function() {
    if (count++ === 100) {
      clearInterval(interval);
      res.end();
    }
    res.write('This is line #' + count + '\n');
  }, 100);
});

After I increased frequency and number of writes it seems to work.

So try:

router.post('/upload', upload.array('upload_data', multerMaxFiles), (req, res, next) => {
 ////some uploading and processing stuff - works nicely
 res.write(JSON.stringify({ 'message': 'File uploaded successfully.' }));// shown to API client nicely
 console.log("what next? " + utilz.inspect(uploaded_file_paths)) //prints file names on console
 next();
});

Answer 3

  //STEP (1)
  //Example simulate upload multiple files with chained api calls. In this example the parameters "idFile" and "arrayidFileExample" are helpful.
  //You should create and send this data from view.
  {
    "idFile": "04fe640f6e4w",                                                                        //id first file                          
    "arrayidFileExample": ["04fe640f6e4w","03g5er4g65erg","g0er1g654er65g4er","0g4er4g654reg654re"]  //simulate idFiles array from view
  }


//STEP (2)
//your upload files api
app.post('/upload', function(req, res) {

    //Express headers, no important in this code
    res.header("Access-Control-Allow-Origin", "*");
    res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");


    let arrayIdFiles = req.body.arrayidFileExample;       //create arrayIdFiles

    let isBlock = false;                                  //flag to block call loop to api
    let countIndex = 0;
    let currentIndex = 0;

    //STEP (3)
    //If "arrayIdFiles" index is not exist, set isBlock to true. Remeber flag to block call loop to api
    for(let count in arrayIdFiles) {
        if(req.body.idFile == arrayIdFiles[count]) {
            console.log("current index --> ", countIndex)
            countIndex++;
            console.log("next index --> ", countIndex)
            if(arrayIdFiles[countIndex] == undefined) {
                isBlock = true;
            }
            break;
        }
        countIndex++;
        currentIndex++;
    }

    //STEP (4)
    //If isBlock is equal false, call get upload api with next idFile. this is simulate "recursive api loop"
    if(isBlock == false) {
        postUploadFile(
          'http://localhost:3500/upload',
          {
            "idFile":arrayIdFiles[currentIndex + 1], //send next idFile from arrayIdFiles
            "arrayidFileExample": arrayIdFiles       //initial arrayIdFiles
          });
    }

    //STEP (6)
    //response json example
    const json = JSON.stringify({
    error:false,
    statusCode: 200,
    body:{
        message:'current id file '+req.body.idFile,
    }
    });

    res.write(json);
    return res.end();
});

    //STEP (5)
    //call "/upload" api post
const postUploadFile = (url = '', body = {}) => {
    return new Promise((resolve, reject)=>{
      axios.post(url, body).then(response => {
          return resolve(response.data);
      }).catch(error => {});
    });
};


//server listen instance
server.listen(3500,() => {
});