How to deal with response from a large file in Node.js?

Question

I am dealing with a large file, where I am making a HTTP request to get the data from. The response is being piped over via stream

fs.createReadStream(pathToFile).pipe(res)

My request function looks like the following

function makeRequest(requestData, IP) {
    var options = {
        host: IP,
        port: 5000,
        path: '/read',
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        }
    };

    var req = http.request(options, function(res) {
        //res.setEncoding('binary');
        var data = [];
        res.on('data', function(chunk) {
            data.push(chunk);
        });

        res.on('end', function() {
            //fs.appendFileSync(fileName, data);
            var binary = Buffer.concat(data);
            var writeStream = fs.createWriteStream(fileName, { "flags": 'a' });
            writeStream.write(binary);
            writeStream.end();
        });

        res.on('error', function(err){
            console.log("Error during HTTP request");
            console.log(err.message);
        });
    });

    req.write(requestData);
    req.end();
}

I believe since its such a large quantity of data, it affects the memory and the server just crashes. How should i deal with this?

This question is based of another question, I had asked before (Reading large files in Node.js from another server)

It depends entirely on what you need to do with the data. If you can do it without needing the entire file contents in memory at the same time, presumably you can manage that with streams. If you need the full contents in memory at the same time, you'll need to ensure that you have enough memory to do that (and don't do it more often than necessary, or in parallel beyond your memory limits, etc.). — T.J. Crowder, Mar 14 '18 at 18:24
From the code in the question (since you never actually said what you're doing), it looks a lot like you're just streaming the response out. Did you try [`pipe`](https://nodejs.org/api/stream.html#stream_readable_pipe_destination_options) and such? Or just writing each `data` chunk out as you receive it? If so, what issue did you run into? — T.J. Crowder, Mar 14 '18 at 18:25

Jonas Wilms · Accepted Answer · 2018-03-14T18:29:32.007

2

Your server does not crash because you are requesting such large files or saving them but rather because you do this:

 var data = [];

You cache the whole 1GB (or whatever size your file has) IN THE RAM! And in a very memory consuming way (why not a buffer?). Instead you should write the data directly to the file and dont cache it:

  var writeStream = fs.createWriteStream(fileName, { "flags": 'a' });

   res.on('data', function(chunk) {
        writeStream.write(chunk);
    });

    res.on('end', function() {
        writeStream.end();
    });

That can be actually simplified to:

 res.pipe(fs.createWriteStream(fileName, { "flags": 'a' }))

Through that only a few chunks stay in the RAM and will be deleted after they were written onto the harddrive.

edited Mar 14 '18 at 18:29

answered Mar 14 '18 at 18:28

Jonas Wilms

132,000
20
149
151

Why not just `.pipe()` it directly from one stream to the other? – jfriend00 Mar 14 '18 at 18:28
@jfriend00 jup, added that, but i think its important to see whats going on behind that oneliner :) – Jonas Wilms Mar 14 '18 at 18:30

How to deal with response from a large file in Node.js?

1 Answers1