0

I have a node api that I'm building that's capable of handling uploads of multipart file types (chunks). This api is based on the Fastify library, and I've already installed the separate Fastify-Multipart library. I've got everything working, including mulitpart file uploads, but part of the requirements of this api is to be able to send requests to another api. In particular, I need to send the file uploads. I don't know what their api is written in, but their multipart file upload api is basically like this:

sdk.files.uploader(location_id, file_size, "filename.jpg", file)
.then(uploader => uploader.start())
.then(file => { console.log(file) })

My code is basically this:

post: async (request, reply) => {

        // Check for file upload
        if((request.headers['content-type'] && request.headers['content-type'].indexOf('multipart/form-data') !== -1) && request.params.id) {

            const multipart = request.multipart(async (field, stream, filename, encoding, mimetype) => {

                console.log(`folderId: ${request.params.id} filename: ${filename}, 
                            field name: ${field}, encoding: ${encoding}, 
                            mime type: ${mimetype}, file length: ${request.headers['content-length']}`)

                try {
                    let uploader = await sdk.files.uploader(request.params.id, Number(request.headers['content-length']), filename, stream)
                    let file = await uploader.start()
                    console.log(file) //Never reaches this point
                }
                catch(e) {
                    console.log(`An error occurred during upload: ${e.message}`)
                    reply.code(500).send()
                }
                //pump(file, fs.createWriteStream(filename))

            }, (error) => {

                if(error) {
                    console.log(`Error uploading file: ${error.message}`)
                    reply.code(500).send()
                } else {
                    console.log('File upload succeeded') //Upload succeeds but it's just in memory
                    reply.code(201).send()
                }
            })

            multipart.on('field', (key, value) => {
                console.log('form-data', key, value)
            })
        }
    }

So basically what I want to do is pass a multipart file stream to this 3rd party api, but doing it this way doesn't seem to be working (when I go on their site, I don't see the file in the folder where it should be). When I look at Activity Monitor on my machine (macOS), I see the node process is consuming 1.2 Gig of memory (roughly the size of the file). Does anyone know a way to do this using Fastify-Multipart (which I believe is based on BusBoy).

u84six
  • 4,604
  • 6
  • 38
  • 65
  • could you add how you register the fastify-multipart plugin? – Manuel Spigolon Jan 28 '20 at 08:12
  • I just register it like this fastify.register(require('fastify-multipart')) – u84six Jan 28 '20 at 16:13
  • I'm looking at ways to pipe it at this point. I see some examples, but all of them are using files that are in the current directory of the node service. What I need is to read the stream from node (via a file upload POST), then pipe it to another api service (that's shown above). Currently, I just wait for the read to complete (the files gets created in the node service directory), then I send it to the 3rd party api when complete. But I don't like that method. I'd rather dispatch the stream to the service if I can. – u84six Jan 28 '20 at 16:16
  • I understand, I think it is possible, I'll give it a try. Anyway I think you need a bit of configuration for the plugin – Manuel Spigolon Jan 28 '20 at 16:34
  • Last question: could you link the API of the `sdk.files.uploader`? – Manuel Spigolon Jan 28 '20 at 16:40
  • I'm sorry I can't do that. The api is private. Anyhow, I'm starting to get the impression that Fastify is becoming obsolete. A bunch of the ecosystem modules don't seem to be supported. For regular file uploads (smaller), fastify-file-upload doesn't install or register with fastify using node 12. Kind of disappointing. – u84six Jan 28 '20 at 17:36
  • Please open an issue to the repo and ask to migrate it in the fastify org. I'm a maintainer of fastify and we already give support to community plugins that need help! You can tag me "@eomm" – Manuel Spigolon Jan 28 '20 at 22:13
  • Anyway ok for the private api, I wanted only check If it can manage the steam I will use a mock for it – Manuel Spigolon Jan 28 '20 at 22:14

1 Answers1

2

I notice that your handler post: async (request, reply) => is async, but you are not calling await, but you are managing the reply in the multipart callback. This can cause issues. Read the promise resolution doc for details.

I would suggest to check the module you are piping the stream since it must use the steam approach and don't save all the chunks to memory.

Here a simple example:

const fastify = require('fastify')({ logger: true })
const pump = require('pump')

fastify.register(require('fastify-multipart'))

fastify.post('/', function (req, reply) { // this function must not be async
  if (!req.isMultipart()) { // you can use this decorator instead of checking headers
    reply.code(400).send(new Error('Request is not multipart'))
    return
  }

  const mp = req.multipart(handler, onEnd)

  mp.on('field', function (key, value) {
    console.log('form-data', key, value)
  })

  function onEnd (err) {
    if (err) {
      reply.send(err)
      return
    }
    console.log('upload completed')
    reply.code(200).send()
  }

  async function handler (field, file, filename, encoding, mimetype) {
    console.log('.....waiting')
    await wait() // testing async function
    pump(file, ConsoleWriter({ highWaterMark: 1000 }))
  }
})

fastify.listen(3000)

function wait () {
  return new Promise(resolve => {
    setTimeout(resolve, 1000)
  })
}

// A writer that manage the bytes
const { Writable } = require('stream')
function ConsoleWriter (opts) {
  return new Writable({
    ...opts,
    write (chunk, encoding, done) {
      console.log({ chunk: chunk.length, encoding })
      setTimeout(done, 500) // slow simulation
    }
  })
}

Call it with:

curl -F file=@"./README.md" -H 'content-type: multipart/form-data' -X POST http://localhost:3000/
Manuel Spigolon
  • 11,003
  • 5
  • 50
  • 73