3

Approximately 10% of the POST or PUT requests I make to my Node.js HTTPS server take 2 minutes to get a response. The other 90% behave as expected.

I have verified with tshark that the body of the request is received by the server, but then it takes 2 minutes to trigger the 'data' and 'end' events of the ServerRequest object.

The requests that experience this latency appear to be random, but when it does happen, the time of the latency between receiving the packet and the 'data' and 'end' events being triggered is always exactly 2 minutes.

For all of these requests I'm posting a small json object that's arriving in a single packet. For the requests experiencing latency, I know the headers are received by my application because the session cookie is parsed immediately. After session authentication, nothing happens until the 'data' and 'end' events are finally emitted 2 minutes later, at which time the body is parsed and saved to my database.

Is this more likely to be an issue with my Node application or a problem with my server, which is an EC2 small instance running Amazon Linux? How is it that it's taking 2 minutes from the time the packet with body of the request arrives until it's passed as a chunk to my application?

Thanks.

Update: I modified parser.onBody in lib/http.js to log 'b' and I can see that this method is not being invoked until the end of the two minute delay. This seems to indicate that the problem is at a lower level than my application. I also changed ec2 instances, but that didn't help.

Andrew Reed
  • 41
  • 4
  • Is it a consistent 2 minutes? How much work is being done on these requests? Any chance it's related to garbage collection? http://stackoverflow.com/questions/5603011/node-js-and-v8-garbage-collection – Timothy Strimple Jan 27 '12 at 07:45
  • Yes, it's exactly two minutes every time. Not much work - I attach a parsed url, parse the session cookie and attach a session object from redis. I've never seen any delays with any of that. I don't think it's garbage collection because I can continue to receive other post requests as normal while one is stuck. – Andrew Reed Jan 27 '12 at 15:22

2 Answers2

3

It is hard to say without knowing exactly what the node application is doing with each request. For instance if it has to talk to another backend service is quite a bit different than just serving an image out of cache that is in ram.

Having said that, the EC2 small instances can be very temperamental. They are heavily shared with other customers, so sometimes they can become weirdly unresponsive. One way to get a feel for this is to run 'top' in the command line of your EC2 instance. Look for the column %st, which is steal. If high, thats bad, switch to a new EC2 instance. (or get a bigger one)

spotman
  • 857
  • 7
  • 11
1

I was listening for 'data' events only after authenticating. I had req.on('data', function(chunk) {req.body += chunk}); in a callback after authenticating, not in the main listener function. Often the payload arrived before I started listening for it.

Andrew Reed
  • 41
  • 4