16

We have a node application running behind Amazon Elastic Load Balancer (ELB), which randomly throws 502 errors when there are multiple concurrent requests and when each request takes time to process. Initially, we tried to increase the idle timeout of ELB to 5 minutes, but still we were getting 502 responses.

When we contacted amazon team, they said this was happening because the back-end is closing the connection with ELB after 5s.

ELB will send HTTP-502 back to your clients for the following reasons:

  • The load balancer received a TCP RST from the target when attempting to establish a connection.
  • The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target.
  • The target response is malformed or contains HTTP headers that are not valid.
  • A new target group was used but no targets have passed an initial health check yet. A target must pass one health check to be considered healthy.

We tried to set our application's keep-alive/timeouts greater than ELB idle timeout (5 min), so the ELB can be responsible for opening and closing the connections. But still, we are facing 502 errors.

js:

var http = require( 'http' );
var express = require( 'express' );
var url = require('url');
var timeout = require('connect-timeout')

const app = express();

app.get( '/health', (req, res, next) => {
  res.send( "healthy" );
});

app.get( '/api/test', (req, res, next) => {
  var query = url.parse( req.url, true ).query;
  var wait = query.wait ? parseInt(query.wait) : 1;
  setTimeout(function() {
    res.send( "Hello!" );
  }, wait );
});

var server = http.createServer(app);
server.setTimeout(10*60*1000); // 10 * 60 seconds * 1000 msecs
server.listen(80, function () {
  console.log('**** STARTING SERVER ****');
});
RAGHU RAMAN
  • 537
  • 4
  • 16

3 Answers3

16

Try setting server.keepAliveTimeout to something other than the default 5s. See: https://nodejs.org/api/http.html#http_server_keepalivetimeout. Per AWS docs, you'd want this to be greater than the load balancer's idle timeout.

Note: this was added in Node v8.0.0

Also, if you're still on the Classic ELB, consider moving to the new Application Load Balancer as based on current experience this seems to have improved things for us a lot. You'll also save a few bucks if you have a lot of separate ELBs for each service. The downside could be that there's 1 point of failure for all your services. But in AWS we trust :)

dandoen
  • 1,647
  • 5
  • 26
  • 44
1

In my case, I needed upgrade nodejs version:

https://github.com/nodejs/node/issues/27363

After that the problem was fixed.

Marcelo Aguiar
  • 161
  • 1
  • 4
  • Welcome to SE! Can you elaborate a bit on why this was a solution to this problem? – Dux Sep 02 '19 at 19:44
  • Based on the link, newer versions of node offer a headersTimout setting which defaults to 40s. If you run a version of node that supports setting this headersTimeout, but you haven't set it, you need to make sure your load balancer timeout and your node server's keepAliveTimeout setting have to be both below 40s. So ELB could have a timeout set to 35s, and the keepAliveTimeout could be set to 36s, and the default headersTimeout of 40s should be OK. Or you could set the headersTimeout to a value higher than your keepAliveTimeout. – wottle Dec 20 '19 at 15:51
  • the TL;DR for this issue is in the [above](https://github.com/nodejs/node/issues/27363) as well as [here](https://adamcrowder.net/posts/node-express-api-and-aws-alb-502/#the-solution). – ab77 Jan 02 '20 at 22:46
-1

Change your server.listen() to this:

const port = process.env.PORT || 3000;
const server = app.listen(port, function() {
  console.log("Server running at http://127.0.0.1:" + port + "/");
});

You can read more about this here.

Matthis Kohli
  • 1,877
  • 1
  • 21
  • 23