Uploading large files to Amazon S3 creates two copies after two minutes

Question

I am trying to upload large files (50 to 60 MB) to amazon S3 bucket from a server written in NodeJS. The server receives the file(s) from a ReactJS frontend. I came across the problem that there are two copies of same file being uploaded to the S3 bucket (although the timestamp differs by two minutes). see the screenshot from my bucket

here's my api written in expressJS/NodeJS :

app.post('/myAPI', (req, res) => {
  
  console.log("------------------------------- In builds API - post request -------------------------------");
  
    console.log(req.method);
    console.log("This device is Android");
    let File = req.files.file;

    //appending the date object to the file name
    let date = JSON.stringify(new Date()).replace(/\"/g, "").replace(/:/g, "-");
    let index = File.name.lastIndexOf(".");
    let newname = [File.name.slice(0, index), date, File.name.slice(index)].join('');
    newname = req.body.project + "/" + newname;
    
    let busboy = new Busboy({ headers: req.headers });
    
    //this event is fired when the data is successfully uploaded
    eventEmitter.on('data_uploaded', function() {
      res.end(`done !!!!!!!!!!!!!!!!!!`);
    });

    busboy.on('finish', function( ) {
      
      let s3bucket = new AWS.S3({
        accessKeyId: IAM_USER_KEY,
        secretAccessKey: IAM_USER_SECRET,
        Bucket: BUCKET_NAME
      });

      s3bucket.createBucket(function () {
        console.log("In create bucket method");
        var params = {
          Bucket: BUCKET_NAME,
          Key: newname,
          Body: File.data
        };
        s3bucket.upload(params, function (err, data) {
          if (err) {
            console.log('error in callback');
            console.log(err);
          }
          
          console.log(data);
          
          eventEmitter.emit('data_uploaded'); 
        });
      });

    });
    
    //piping the request to writable stream busboy
    req.pipe(busboy);
  }

});

here's what my console looks like

.. everything two times.

I am checking into my browser's network tab and the request sent is only one..

and though the response is received only once as this:

What am I doing wrong.. please help me out here ! Thanks a million in advance.

( I wrote this API using this tutorial by Keith Weaver ).

score 0 · Accepted Answer · edited Aug 02 '18 at 04:46

The req object expires after the default timeout of 2 minutes, which was the cause of creation of a new req object and a new API hit.

Overriding the timeout key on server instance solved my problem :

let server = app.listen(port, () => { console.log(Started up at port ${port}); });

server.timeout = 600000; //6 lac milliseconds = 10 minutes, default is 2 minutes

module.exports = {app};

Thanks to this answer by SomeKittens.

Uploading large files to Amazon S3 creates two copies after two minutes

1 Answers1