Getting TCP._onclose intermittently on http transfer of large file

Question

I am using jquery's ajax call to transfer a 10GB file to my node.js server. I was pleased that my first tests succeeded, but in subsequent tests, I'm getting a call to TCP._onclose on the server which aborts the transfer. I'm trying to avoid doing manual file I/O to chop up the file because the file I/O reads are killing my transfer rate. So, my questions are two-fold:

1) Is the TCP._onclose to be expected when transferring a file this large (or do I have a bug somewhere that needs tracking down)?

2) If I have to dice up the file manually to send smaller block, what is a reasonable block size and is there a more efficient way to dice it up than using file.slice(..) followed by FileReader.readAsBinaryString(..) and sending the data on the reader onload function?

Here is my current client-side code:

$( "#streamUploadButton" ).click( function() {        
    lastPlace = 0;
    startTime = lastTime = Date.now();
    $.ajax( {
        xhr: function() {
            var xhr = new XMLHttpRequest();
            xhr.upload.addEventListener( "progress", function( evt ) {
                if ( evt.lengthComputable ) {
                    var place = evt.loaded;
                    var percentComplete = place / evt.total * 100;
                    updateStats( $( "#uploadStreamingRate" ), place,
                        $( "#uploadProgress" ), percentComplete );
                    lastPlace = place;
                    lastTime = Date.now();
                }
            }, false );
            return xhr;
        },
        url: "/httpUpload",
        type: "POST",
        processData: false,
        contentType: false,
        data: createFileFormData( $( "#uploadFileInput" ) ),
        success: function() {
            reportComplete( $( "#uploadStreamingRate" ), $( "#uploadProgress" ) );
        },
        error: function( error ) {
            console.error( "HttpUpload failed: " + error.responseText );
        }
    } );
} );

function createFileFormData( $fileInput ) {
    var formData = new FormData();
    file = $fileInput[ 0 ].files[ 0 ];
    formData.append( "file", file );
    return formData;
}

And on the node.js with express server:

app.post( "/httpUpload", function( req, res ) {
    res.send( "Done!" );
    var file = req.files.file;
    fs.renameSync( file.path, __dirname + '/temp/' + file.name );
    console.log( "Received post request");
} );

Here is the error message that I get in the server console:

Error: Request aborted
at IncomingMessage.onReqAborted (node_modules\connect-multiparty\node_modules\multiparty\index.js:183:17)
at emitNone (events.js:67:13)
at IncomingMessage.emit (events.js:166:7)
at abortIncoming (_http_server.js:267:11)
at Socket.serverSocketCloseListener (_http_server.js:280:5)
at emitOne (events.js:82:20)
at Socket.emit (events.js:169:7)
at TCP._onclose (net.js:469:12)

are you using express.js (I guess yes). Maybe just adding a higher server.timeout helps a bit. I think generally, your upload takes too long and the server just abortes the request due to the server.timeout limit. And according your error message, you use "multiparty" a fork of "formidable". This module is using the timeout value -> http://stackoverflow.com/questions/23925284/how-to-modify-the-nodejs-request-default-timeout-time/23926952#23926952 or https://github.com/felixge/node-formidable/issues/304 — Fer To, Oct 05 '15 at 16:12
You are my hero. That fixed it! If you would like to repost your comment as an answer, I'll gladly accept it. :o) — Eric, Oct 05 '15 at 17:13
here you go :) if you use it for some sort of "productive" case I would recommend to look up into proper solutions, e.g. HTML5 File API or plugins like http://www.resumablejs.com/ — Fer To, Oct 07 '15 at 08:16

score 1 · Accepted Answer · answered Oct 07 '15 at 08:14

Ok, as Eric accepted the answer. For all future users the KISS solution for this issue would be to increase your timeout value of your server configuration.

For example 12000ms = 12s would be the max. time after the request is aborted.

server.timeout = 120000

Additionally, I would recommend to have some sort of "progress" feedback for the user if you're uploading such big files, e.g. the basic loading circle.

Getting TCP._onclose intermittently on http transfer of large file

1 Answers1