4

I'm trying to receive a multipartform-data stream which might include various files and fields and to write the files to a directory (uWebsockets.js server). I have this code:

    let boundary = null;
    let fields = [];
    let streams = [];
    let keep = false;
    res.onData((chunk, isLast) => {
        const buff = Buffer.concat([Buffer.from(chunk)]).toString('binary').split('\r\n');
        if (!boundary) {
            boundary = buff[0];
        }
        for (let i = 0; i < buff.length; i++) {

            const line = buff[i];
            if (i > 0 && i < buff.length - 1 && line == '') {
                keep = true;
                continue;
            }
            if (line == boundary || line == boundary + '--') {
                keep = false;
                if (streams[fields.length - 1]) {
                    streams[fields.length - 1].end();
                }
            }
            if (line == boundary) {
                fields[fields.length] = {};
            }

            if (line.includes('Content-Disposition')) {
                if (line.includes('filename="')) {
                    fields[fields.length - 1].filename = getFilename(line);
                    fields[fields.length - 1].type = 'file';
                    fields[fields.length - 1].path = path.resolve(options.uploadPath + fields[fields.length - 1].filename);
                    streams[fields.length - 1] = fs.createWriteStream(
                        path.resolve(options.uploadPath + fields[fields.length - 1].filename)
                    );
                } else {
                    fields[fields.length - 1].type = 'field';
                }
                fields[fields.length - 1].name = getField(line);
            }
            if (line.includes('Content-Type')) {
                fields[fields.length - 1].contentType = line.split('Content-Type: ')[1];
            }

            if (keep == true) {
                if (fields[fields.length - 1].filename) {
                    streams[streams.length - 1].write(Buffer.from(line + "\r\n", 'binary'));
                } else {
                    fields[fields.length - 1].value += line;
                }
            }
        }

        if (isLast) {
            console.log(fields);
        }
    });

It works except that uploaded images are corrupted and are cut in randomly (Not the same in every image, some are totally corrupted and some are perfectly fine). Could someone point out what is wrong with it?

Thanks in advance :)

Daniel Shlomo
  • 105
  • 1
  • 2
  • 17
  • 1
    Have you looked into Mutler: https://www.npmjs.com/package/multer Form-data: https://www.npmjs.com/package/form-data I have created a Node endpoint that consumes `multipartform-data` submission. – mmason33 Aug 27 '20 at 01:51
  • 1
    I can share the code if you want. As far as your code I commend you for trying to do it manually. – mmason33 Aug 27 '20 at 01:52
  • 1
    @mmason33 Yes please, if you could share it with me then it will be great! Thanks – Daniel Shlomo Aug 27 '20 at 08:30
  • @mmason33 Mutler is a Express middleware right? if so then it won't work since i'm not using Express. I'm using uWebsocket.js as a replacement to Express. – Daniel Shlomo Aug 27 '20 at 08:34

3 Answers3

2

same situation here, i am using uwebsocket.js for uploading mp4 file, i try your code to upload mp4, it's uploaded but cannot be play. then i try to compare original file with uploaded file, the file size has change a litle bit.

then i realize in your code :

streams[streams.length - 1].write(Buffer.from(line + "\r\n", 'binary'));

when split by "\r\n" last of array should not have "\r\n" than i add some condition below :

if (i==buff.length-1)
   streams[streams.length - 1].write(Buffer.from(line, 'binary'));
else {
   if (i+1<buff.length) {
      if (buff[i+1]==boundary + '--')
         streams[streams.length - 1].write(Buffer.from(line, 'binary'));
      else
         streams[streams.length - 1].write(Buffer.from(line + "\r\n", 'binary'));
   }
}

and now my uploaded mp4 play well.

thanks....

sorry for my bad english

sehari3kali
  • 36
  • 3
1

probability problem is max length. change this value on uwebsocket.js. best way is form data for upload file.

change this:

maxPayloadLength: 512

more detail: https://unetworking.github.io/uWebSockets.js/generated/interfaces/websocketbehavior.html#maxpayloadlength

eay
  • 176
  • 6
  • 1
    That's for websocket payload. i use uWebsocket as a replacement for Express. – Daniel Shlomo Aug 27 '20 at 08:30
  • 1
    By the way, if i concat all the chunks and then write all the files then everything works. the only problem is that it won't work with very large files. that's why i'm trying to write the files directly from stream. – Daniel Shlomo Aug 27 '20 at 08:44
  • 1
    Is your goal for someone to upload a file and for someone else to be instantly informed? if is it like that you use form data and web socket is sends a key to you has (temp on redis or like some thing ) (example: mysite/upload/:key).https://stackoverflow.com/a/11081396/13351032 – eay Aug 28 '20 at 06:46
  • 1
    no, even though there's "websocket" im the name of uWebsocket.js, it's also an http server (like express). I'm trying to read a multipart form stream and write the uploaded files. I don't use a Websocket in this code. – Daniel Shlomo Aug 31 '20 at 01:37
1

Have you tried formidable? I use it to parse my multipartform-data streams.

Alhassan Raad
  • 179
  • 11
  • Can it parse from stream chunks? Or it needs the stream to finish and then parse it? – Daniel Shlomo Sep 01 '20 at 00:15
  • Since no one is yet to find a solution and i need to give the bounty reward then i'll give it to you (also because you don't have a lot of points). – Daniel Shlomo Sep 02 '20 at 04:14
  • Yes, you can parse from stream chunks. – Alhassan Raad Sep 02 '20 at 11:03
  • It doesn't write directly from stream chunks. It concats the chunks and then parse. So it's not what i need. https://github.com/node-formidable/formidable/blob/5110ef8ddb78501dcedbdcb7e2754d94abe06bc5/src/Formidable.js#L221 – Daniel Shlomo Sep 03 '20 at 04:08