Getting ERR_FS_FILE_TOO_LARGE while using unirest file send with put

Question

I am using unirest to upload a file like so

 unirest.put(fullUri)
    .auth({
      user: self.userName,
      pass: self.password
    })
    .header('X-Checksum-Sha1', sha1Hash)
    .header('X-Checksum-Md5', md5Hash)
    .send(fs.readFileSync(filePath))
    .end(function (response) {

This works fine for smaller files but for large files I get ERR_FS_FILE_TOO_LARGE error. I have already tried max_old_space_size without success. Looks like I can fix this by streaming the file but I can't find an api to do that in unirest js library.

What are your memory limits? Do you have enough ram to load the entire file into ram? — Tschallacka, Jul 13 '20 at 13:03
I know you're looking for an answer with unirest, but unirest is kind of deprecated. I recommend using axios instead. — Devchris, Jul 19 '20 at 15:04

score 3 · Answer 1 · answered Jul 13 '20 at 13:15

Looks like this is an issue with form-data From GitHub Issues

It turns out that the unirest are using the NPM module form-data, and the form-data module requires that if it received a stream that not fs.ReadStream, we need to provide the file information by additional.

Example:

form.append( 
  'my_file', 
  stream,
  {
    filename: 'bar.jpg', 
    contentType: 'image/jpeg', 
    knownLength: 19806,
  },
)

See: https://github.com/form-data/form-data#void-append-string-field-mixed-value--mixed-options-

score 0 · Answer 2 · answered Jul 14 '20 at 07:04

0

Streaming files with unirest is available via the .attach method:

 unirest.put(fullUri)
    .auth({
      user: self.userName,
      pass: self.password
    })
    .header('X-Checksum-Sha1', sha1Hash)
    .header('X-Checksum-Md5', md5Hash)
//    .send(fs.readFileSync(filePath))
    .attach('filename', fs.createReadStream(filePath))
    .end(function (response) {

answered Jul 14 '20 at 07:04

Avraham

928
4
12

I already tried this and the server doesn't accept it. This is only for multipart/form-data I think – parapura rajkumar Jul 14 '20 at 13:13
There is another way to stream data over HTTP, using `Transfer-Encoding: chunked`, which is supported by the `request` package, but `unirest` doesn't seem to support it. Does your server accept any streaming method at all? – Avraham Jul 14 '20 at 13:35

score -1 · Answer 3 · answered Jul 17 '20 at 15:27

I can't find an API to do that in unirest js library.

That's because there is none: https://github.com/Kong/unirest-nodejs/issues/49:

You can use the underlying request library to do streaming if you want, I am open to a pull request either on this branch or the 1.0 version to add streaming.

Issue is still open.

But from this issue and from the source code you can find out that end() returns request (see https://github.com/request/request)

Unirest.request = require('request')

...

  end: function (callback) {
    ...
    Request = Unirest.request($this.options, handleRequestResponse)
    Request.on('response', handleGZIPResponse)
    ...
    return Request
  }

and from request's source code you can find out that no actual request is sent yet (it's defered). So you can hack into it. And use it's API instead:

const request = unirest.put(constants.server2)
  .auth({
    user: self.userName,
    pass: self.password
  })
  .header('X-Checksum-Sha1', sha1Hash)
  .header('X-Checksum-Md5', md5Hash)
  // .send(fs.readFileSync(filePath))
  .end(...)

fs.createReadStream(filePath).pipe(request) // just pipe it!

As a side note: unirest is based on request, request is deprecated now. So... maybe you need to steer away from unirest.

Getting ERR_FS_FILE_TOO_LARGE while using unirest file send with put

3 Answers3

Example: