Load Balancing with Node and Heroku

Question

I have a web app that accepts api requests from an ios app. My web app is hosted on Heroku using their free dyno which is able to process 512 mb of data per request. Because node is a single threaded application this will be a problem once we start getting higher levels of traffic from the ios end to the web server. I'm also not the richest person in the world so i'm wondering if it would be smart to create another free heroku app and use a round robin approach to balance the load received from the ios app?

I just need to be pointed into the right direction. Vertical scaling is not really an option financially.

Answer 1

I'm the Node.js platform owner at Heroku.

You may be doing some premature optimization. Node.js, on our smallest 1X size (512MB RAM), can handle hundreds of simultaneous connections and thousands of requests per minute.

If your iOS app is consistently maxing that out, it may be time to consider monetization!

Answer 2

As mentioned by Daniel it's against Heroku rules. Having said that there are probably other services that would allow you to do that. One way to approach this problem is to use cluster module with ZeroMQ (you need to have ZeroMQ installed before using the module - see module description).

var cluster = require('cluster');
var zmq = require('zmq');

var ROUTER_SOCKET = 'tcp://127.0.0.1:5555';
var DEALER_SOCKET = 'tcp://127.0.0.1:7777';

if (cluster.isMaster) {
  // this is the main process - create Router and Dealer sockets
  var router = zmq.socket('router').bind(ROUTER_SOCKET);
  var dealer = zmq.socket('dealer').bind(DEALER_SOCKET);

  // forward messages between router and dealer
  router.on('message', function() {
    var frames = Array.prototype.slice.cal(arguments);
    dealer.send(frames);
  });

  dealer.on('message', function() {
    var frames = Array.prototype.slice.cal(arguments);
    router.send(frames);
  });

  // listen for workers processes to come online
  cluster.on('online', function() {
    // do something with a new worker, maybe keep an array of workers
  });

  // fork worker processes
  for (var i = 0, i < 100; i++) {
    cluster.fork();
  }
} else {
  // worker process - connect to Dealer
  let responder = zmq.socket('rep').connect(DEALER_SOCKET);

  responder.on('message', function(data) {
    // do something with incomming data
  })
}

This is just to point you in the right direction. If you think about it you can create a script with a parameter that will tell it if it's a master or a worker process. Then on the main server run it as is, and on additional servers run it using worker flag which will force it to connect to the main dealer.

Now your main app needs to send the requests to the router, which will be later forwarded to the worker processes:

var zmq = require('zmq');
var requester = zmq.socket('req');

var ROUTER_SOCKET = 'tcp://127.0.0.1:5555';

// handle replies - for example completion status from the worker processes
requester.on('message', function(data) {
  // do something with the replay
});

requester.connect(ROUTER_SOCKET);

// send requests to the router
requester.send({
  // some object describing the task
});

Answer 3

So first off, as the other replies have pointed out, running two copies of your app to avoid Heroku's limits violates their ToS, which may not be a great idea.

There is, however, some good news. For starters (from Heroku's docs):

The dyno manager will restart your dyno and log an R15 error if the memory usage of a:

• free, hobby or standard-1x dyno reaches 2.5GB, five times its quota.

As I understand it, despite the fact that your dyno has 512mb of actual RAM, it'll swap out to 5x that before it actually restarts. So you can go beyond 512mb (as long as you're willing to pay the performance penalty for swapping to disk, which can be severe).

Further to that, Heroku bills by the second and allows you to scale your dyno formation up and down as needed. This is fairly easy to do within your own app by hitting the Heroku API – I see that you've tagged this with NodeJS so you might want to check out:

• Heroku's node client
• the very-barebones-but-still-functional toots/node-heroku module

Both of these modules allow you to scale up and down your formation of dynos — with a simple heuristic (say, always have a spare 1X dyno running), you could add capacity while you're processing a request, and get rid of the spare capacity when api requests aren't running. Given that you're billed by the second, this can end up being very inexpensive; 1X dynos work out to something like 5¢ an hour to run. If you end up running extra dynos for even a few hours a day, it's a very, very small cost to you.

Finally: there are also 3rd party services such as Adept and Hirefire (two random examples from Google, I'm sure there are more) that allow you to automate this to some degree, but I don't have any experience with them.

Answer 4

You certainly could, I mean, programatically - but that would bypass Heroku's TOS:

4.4 You may not develop multiple Applications to simulate or act as a single Application or otherwise access the Heroku Services in a manner intended to avoid incurring fees.

Now, I'm not sure about this:

Because node is a single threaded application this will be a problem once we start getting higher levels of traffic from the ios end to the web server.

There are some threads discussing that, with some interesting answers:

Clustering Node JS in Heavy Traffic Production Environment

How to decide when to use Node.js?

Also, they link to this video, introducing Node.js, which talks a bit about benchmarks:

Introduction of Node JS by Ryan Dahl