Learning Node - Book - Pages 16-18 - Two Examples

Question

There are two examples in between these pages 16 and 18.
Example 1.3 is a server app.
Example 1.4 is a client app doing GET requests to the server.

When I run the two examples (at the same time) I notice some quite weird behavior
in the client. All requests are executed (i.e. the for loop in the client completes)
but the callbacks of only 5 of them get called. The client doesn't exit and also
doesn't error out. And just no more callbacks are called.

Any ideas what might be happening or how I can troubleshoot this further?

Note: I am running Node.js v0.10.20 on Windows 7.

Server:

var http = require('http');
var fs = require('fs');

// write out numbers
function writeNumbers(res) {
    var counter = 0;
    // increment, write to client
    for (var i = 0; i<100; i++) {
        counter++;
        res.write(counter.toString() + '\n');
    }
}

// create http server
http.createServer(function (req, res) {
    var query = require('url').parse(req.url).query;
    var app = require('querystring').parse(query).file;

    // content header
    res.writeHead(200, {'Content-Type': 'text/plain'});

    if (!app){
        res.end();
        console.log('No file argument found in query string.');
        return;
    }else{
        app = app + ".txt";
    }

    // write out numbers
    writeNumbers(res);

    // timer/timeout to open file and read contents
    setTimeout(function() {
        console.log('Opening file: ' + app + '.');
        // open and read in file contents
        fs.readFile(app, 'utf8', function(err, data) {
            res.write('\r\n');
            if (err)
                res.write('Could not find or open file ' + app + ' for reading.\r\n');
            else {
                res.write(data);
            }
            // response is done
            res.end();
        });
    },2000);
}).listen(8124);
console.log('Server running at 8124');

Client:

var http = require('http');

var N = 200;

// The URL we want, plus the path and options we need
var options = {
    host: 'localhost',
    port: 8124,
    path: '/?file=automatic',
    method: 'GET'
};

var callback_function = function(response) {
    // finished? ok, write the data to a file
    console.log('got response back');
};

for (var i = 1; i <= N; i++) {
    // make the request, and then end it, to close the connection
    http.request(options, callback_function).end();
    console.log('done with call # ' + i);
}

Screenshot while running both

--- Experiment Done ---

If I lower N to 10 and also if I do a
global "var i = 1" and then do this thing:

function schedule(){
    http.request(options, callback_function).end();
    console.log('done with call ' + i);
    i++;
    if (i<=N){
        setTimeout(function(){
            schedule();
        }, 1000);
    }
}

schedule();

instead of the loop in the client, I get similar behavior.
I guess that's what Milimetric meant by "sleep" i.e. just
to make sure I don't hit the server too quickly with too
many simultaneous requests.

But the behavior is not fully identical, it takes several mins to print 'got response back' on the second set of 5 requests and then another maybe 5-6 mins for the client to exit. Still, all that does look weird to me.

C:\PERSONAL\NODE_TEST>node test004.js
done with call 1
got response back
done with call 2
got response back
done with call 3
got response back
done with call 4
got response back
done with call 5
got response back
done with call 6
done with call 7
done with call 8
done with call 9
done with call 10
got response back
got response back
got response back
got response back
got response back

C:\PERSONAL\NODE_TEST>

Why don't you show the actual code? Or do you expect answers only from people that happen to have this book in front of them? — p.s.w.g, Nov 27 '13 at 21:34
@zerkms No no ;) Just cannot get this example working for 2-3 days and decided to ask here. Let me post the code which I reworked a bit actually while trying to troubleshoot it. — peter.petrov, Nov 27 '13 at 21:38
can you try putting a sleep of like a second after your `http.request` call? Probably lower N to like 10 as well, and let us know what you get. That's very interesting. If I have some time, I'll try this out — Milimetric, Nov 27 '13 at 21:58
What happens if you eliminate the delay in the server (keeping the client delay)? I wonder if the client thinks it is timing out — SheetJS, Nov 28 '13 at 01:15
@Nirk If I decrease the delay in the server from 2000 to 2 milliseconds, I am seeing the same behavior as described in the "Additional Experiment" section. — peter.petrov, Nov 28 '13 at 10:09

score 1 · Accepted Answer · edited May 23 '17 at 11:56

The problem is that the client doesn't consume the response body sent by the server, so the connection remains (half) open and the http agent only allows 5 concurrent requests per client by default, causing it to hang after 5 requests. The connection will eventually timeout, causing the next 5 requests to be processed.

node.js http.get hangs after 5 requests to remote site

Change your callback function to consume any data sent down the response.

var callback_function = function(response) {
    // finished? ok, write the data to a file
    console.log('got response back');
    response.on('data', function () {});
};

That does look much better. Thank you for the answer and for the link too. — peter.petrov, Nov 29 '13 at 18:55

Learning Node - Book - Pages 16-18 - Two Examples

1 Answers1