NodeJs API sending blank response while converting larger files

Question

I am currently working on a piece of code that uploads mp4 videos from my localhost onto a server. The basic thing is that if the video is a .mp4 it is directly uploaded, else it is converted to .mp4 and then uploaded. I'm using the video converter using handbrake-js.

All works fine except for a tiny part. When the file isn't that huge, say less than 70-80 Mb, it works like a charm. But the problem is with larger files. Even though I am explicitly calling the res.end / res.send in the .on(end) callback, I receive some blank response in my angular controller, even before the conversion has finished. I have noticed it happens at around 30 to 40% of the conversion. It has a readystate which is equal to XMLHttpRequest.DONE and also status = 200.

Here is the Node side code:

try {
    if (fs.existsSync(uploadPath + filename.substring(0, filename.lastIndexOf('.')) + '.mp4')) {
        res.end('<b><i>' + filename + '</i></b> already exists in the directory.');
    }
    else {
        const fstream = fs.createWriteStream(path.join(cfg.tempStoragePath, filename));
        file.pipe(fstream);
        console.log("\nfile_type: " + file_type);
        filename = filename.substring(0, filename.lastIndexOf('.'));

        // On finish of the copying file to temp location
        fstream.on('close', () => {
            hbjs.spawn({
                input: cfg.tempStoragePath + filename + '.' + file_type,
                output: uploadPath + filename + '.mp4'
            })
                .on('error', err => {
                    // invalid user input, no video found etc
                    console.log('error! No input video found at: \n: ' + cfg.tempStoragePath + filename + '.' + file_type);
                    res.send('Conversion of the file, <b><i>' + filename + '</i></b>, from <b>.' + file_type + '</b>' + ' to <b>.mp4</b> failed because no input video was found at: \n: ' + cfg.tempStoragePath + filename + '.' + file_type);
                })
                .on('progress', progress => {
                    progress_percent = (Number(progress.percentComplete) * 2 <= 100) ? Number(progress.percentComplete) * 2 : 100;
                    eta = progress.eta.split(/[a-zA-Z]/);
                    minutes = ((+eta[0]) * 60 + (+eta[1])) / 2;
                    console.log('Percent complete: %d, ETA: %d ///// %s ==> mp4', progress_percent, minutes, file_type);
                })
                .on('end', end => {
                    console.log('Conversion from .' + file_type + ' to .mp4 complete.');
                    //delete the temp file
                    fs.unlink(cfg.tempStoragePath + filename + '.' + file_type);
                    let new_path = uploadPath + filename + '.mp4';
                    let stat = fs.statSync(new_path);
                    console.log(`Upload of '${filename}' finished`);
                    if(Number(progress_percent) === Number(100))
                        res.send('The file, <b><i>' + filename + '</i></b>, has been converted from <b>.' + file_type + '</b>' + ' to <b>.mp4</b> complete.');
                    })
                });
            }
    } 

    catch (err) {
        res.end(err);
    }

Following is part of my angular controller:


request = new XMLHttpRequest();
request.onreadystatechange = function () {
    if (request.readyState === XMLHttpRequest.DONE && request.status === 200) {
        showConversionModal('<p>' + request.responseText + '</p>', 'done');
    }
};
showSomeModal('something');
request.open("POST", client.clientHost + ":" + client.clientPort + "/uploadVideoService");
formData = new FormData();
formData.append("file", files[0], files[0].name);
request.send(formData);

NOTE : All the data which is do a console.log() inside the node is as expected. And even the res.end/send works fine for smaller files which take lesser time. But the problem arises only for those which conversion takes longer than the smaller files.

Also, the if loop which checks for existing file scenario doesn't work as expected for these larger files. I thought at least that should work, because it doesn't even get into the handbrake part. But that is not the case.

And in the browser I get this error:

Failed to load resource: net::ERR_CONNECTION_RESET

which points to the request.send(formData); line, and I have also tried almost all solution from this SO article, with no good effect. But still, the conversion happens fine.

P.S.: Also note that the conversion and upload happens without a problem even for the larger files, it's just the response I'm receiving on the client side that has been my headache.

UPDATE: I tried using the debugger in VS Code, and saw that the breakpoints are rightly hitting the res.end() inside the if loop which checks for existing file scenario, but for some strange reason, the angular controller isn't reacting to it. And this happens only for the larger files.

score 0 · Accepted Answer · answered May 23 '19 at 11:01

I figured it out myself after a long time. Turns out there's something like a buffer in Busboy.

Here, I used to link the busboy and express like this:

app.use(busboy());

Just setting the field called highWaterMark to an arbitrary size which would be greater than the file size did the trick.

app.use(busboy({
    highWaterMark: 2048 * 1024 * 1024, // Set buffer size in MB
}));

I don't know what might have caused this problem, or what the new field does that solves it, but, it just works is all that I know. It would be helpful if someone could elaborate on this a bit.

NodeJs API sending blank response while converting larger files

1 Answers1