How to write a file from an ArrayBuffer in JS

Question

I am trying to write a file uploader for Meteor framework. The principle is to split the fileon the client from an ArrayBuffer in small packets of 4096 bits that are sent to the server through a Meteor.method.

The simplified code below is the part of the client that sends a chunk to the server, it is repeated until offset reaches data.byteLength :

// data is an ArrayBuffer
var total = data.byteLength;
var offset = 0;

var upload = function() {
  var length = 4096; // chunk size

  // adjust the last chunk size
  if (offset + length > total) {
     length = total - offset;
  }

  // I am using Uint8Array to create the chunk
  // because it can be passed to the Meteor.method natively
  var chunk = new Uint8Array(data, offset, length);

  if (offset < total) {
     // Send the chunk to the server and tell it what file to append to
     Meteor.call('uploadFileData', fileId, chunk, function (err, length) {
        if (!err) {
          offset += length;
          upload();
        }
     }
  }
};
upload(); // start uploading

The simplified code below is the part on the server that receives the chunk and writes it to the file system :

var fs = Npm.require('fs');
var Future = Npm.require('fibers/future');

Meteor.methods({
  uploadFileData: function(fileId, chunk) {
    var fut = new Future();
    var path = '/uploads/' + fileId;

    // I tried that with no success
    chunk = String.fromCharCode.apply(null, chunk);

    // how to write the chunk that is an Uint8Array to the disk ?
    fs.appendFile(path, chunk, 'binary', function (err) {
      if (err) {
        fut.throw(err);
      } else {
        fut.return(chunk.length);
      }
    });
    return fut.wait();
  }
});

I failed to write a valid file to the disk, actually the file is saved but I cannot open it, when I see the content in a text editor, it is similar to the original file (a jpg for example) but some characters are different, I think that could be an encoding problem as the file size is not the same, but I don't know how to fix that...

can you pass a blob instead? they usually come out in node as Buffers, which fs.AppendFile() handles. — dandavis, Jul 23 '15 at 08:18
@dandavis Actually you gave me the half of the answer, see the solution was simple after all, Thank you ! — Karl.S, Jul 23 '15 at 09:51

score 36 · Accepted Answer · edited May 03 '20 at 17:47

36

Saving the file was as easy as creating a new Buffer with the Uint8Array object :

// chunk is the Uint8Array object
fs.appendFile(path, Buffer.from(chunk), function (err) {
    if (err) {
      fut.throw(err);
    } else {
      fut.return(chunk.length);
    }
});

edited May 03 '20 at 17:47

Etheryte

24,589
11
71
116

answered Jul 23 '15 at 09:51

Karl.S

2,294
1
26
33

7

new Buffer(chunk) is deprecated use Buffer.from(chunk) – PRAJIN PRAKASH Mar 07 '20 at 07:15

score 27 · Answer 2 · edited Nov 21 '21 at 04:49

27

Building on Karl.S answer, this worked for me, outside of any framework:

fs.appendFileSync(outfile, Buffer.from(arrayBuffer));

edited Nov 21 '21 at 04:49

Brendan Nee

5,087
2
33
32

answered Oct 16 '17 at 21:12

toms

1,760
1
16
17

11

new Buffer is deprecated. Use Buffer.from(arrayBuffer) instead. – tornord Dec 13 '19 at 15:00
1

Using synchronous version of I/O functions is not recommended since it block the event loop, unless you have a reason to do so. – Karl.S Jul 07 '21 at 21:13

score 2 · Answer 3 · answered Oct 27 '16 at 16:27

Just wanted to add that in newer Meteor you could avoid some callback hell with async/await. Await will also throw and push the error up to client

Meteor.methods({
  uploadFileData: async function(file_id, chunk) {
    var path = 'somepath/' + file_id; // be careful with this, make sure to sanitize file_id
    await fs.appendFile(path, new Buffer(chunk));
    return chunk.length;
  }
});

How to write a file from an ArrayBuffer in JS

3 Answers3

Linked