7

Im using DropBox API to upload files. To upload the files to dropbox I am going through the following steps:

  1. First upload file from form to a local directory on the server.
  2. Read File from local directory using fs.createReadStream
  3. Send file to Dropbox via the dropbox API.

The issue:

For some reason fs.createReadStream takes absolute ages when reading and uploading a large file. Now the file I'm trying to upload is only 12MB which is not a big file and it takes approximately 18mins to upload/process a 12MB file.

I don't know where the issue is either it's in createReadStream or dropbox api code.

It works with files of size within kb.

My Code:

let options = {
    method: 'POST',
    uri: 'https://content.dropboxapi.com/2/files/upload',
    headers: {
        'Authorization': 'Bearer TOKEN HERE',
        'Dropbox-API-Arg': "{\"path\": \"/test/" + req.file.originalname + "\",\"mode\": \"overwrite\",\"autorename\": true,\"mute\": false}",
        'Content-Type': 'application/octet-stream'
    }, 

      // I think the issue is here.
      body: fs.createReadStream(`uploads/${req.file.originalname}`)
};

 rp(options)
    .then(() => {
        return _deleteLocalFile(req.file.originalname)
    })
    .then(() => {
        return _generateShareableLink(req.file.originalname)
    })
    .then((shareableLink) => {
        sendJsonResponse(res, 200, shareableLink)
    })
    .catch(function (err) {
        sendJsonResponse(res, 500, err)
    });

Update:

const rp = require('request-promise-native');
Skywalker
  • 4,984
  • 16
  • 57
  • 122
  • what is rp ? the problem is probably more how it handles streams rather than the stream itself – laurent Jan 02 '18 at 18:38
  • @laurent thank you for the comment. I've updated my code which explains what rp is. Basically its a `request` promise library. I've been trying to fix this for days. It seems crazy that it takes **18mins** to upload a **12mb** file. – Skywalker Jan 02 '18 at 18:41
  • 3
    If you wish to stream the request documentation advises to directly pipe: fs.createReadStream('file.json').pipe(request.post('targetSite',optionsAndHeaders)). However there are more "modern" http client nodejs libraries. – laurent Jan 02 '18 at 18:45
  • Actually you even have a DropBox sdk with an upload example https://github.com/dropbox/dropbox-sdk-js/blob/master/examples/javascript/node/upload.js – laurent Jan 02 '18 at 18:52
  • Try watching memory usage of the node process while it's running. Perhaps there's a memory leak somewhere. – David Knipe Jan 04 '18 at 20:22
  • You can always compare your time result with upload feature available in official Dropbox web-app. – Adam Płócieniak Jan 06 '18 at 00:46

2 Answers2

3

I had an experience similar to this issue before and after a large amount of head scratching and digging around, I was able to resolve the issue, in my case anyway.

For me, the issue arose due to the default chunking size for createReadStream() being quite small 64kb and this for some reason having a knock on effect when uploading to Dropbox.

The solution therefore was to increase the chunk size.

// Try using chunks of 256kb
body: fs.createReadStream(`uploads/${req.file.originalname}`, {highWaterMark : 256 * 1024});
Glen
  • 1,178
  • 10
  • 19
0

https://github.com/request/request#streaming

I believe you need to pipe the stream to the request.

see this answer: Sending large image data over HTTP in Node.js

Rob Bruhn
  • 1
  • 1
  • 3