4

I'm using NodeJS and amqplib to build a simple job queueing library (somewhat similar to Jackrabbit) to be used in a service that should parse pretty large feeds containing information about a lot of different events.

After the information is parsed, it gets distributed to thousands of queues (the queue is the smallest unit where message order has to be respected) and then consumed by my queueing library.

The question is: how do I follow the '1 connection per app, 1 channel per thread, 1 consumer per channel' guideline in this case? It would cause thousand of processes to be spawned, wasting huge amounts of memory and resources.

Note: The number of messages in each of the queues is not very high: about 1-2 msg/sec at most

Towerman
  • 185
  • 1
  • 4
  • The guidelines I'm talking about are mentioned in this post: http://stackoverflow.com/questions/10407760/is-there-a-performance-difference-between-pooling-connections-or-channels-in-rab – Towerman Dec 01 '16 at 13:10

1 Answers1

12

I'm assuming you know that NodeJS is single-threaded when running your code, which is where your question comes from...

In languages that are thread-based, the "1 channel per thread" rule is important because the language will block the thread on which the channel / consumer is sitting. You'll get to some kind of consume call, and the thread will block, waiting for a message to appear.

This is why you need to have 1 thread per channel, in languages that support threads.

In NodeJS, though, the call for a subscriber to consume messages is non-blocking.

This means you can safely throw out the 1 channel per thread concept.

My own NodeJS and RabbitMQ code will often have hundreds of channels open in a single instance of my application.

Channels are cheap and easy to open. Setting up a message producer or consumer is also cheap. The real cost is 1) in the connection, and 2) doing the actual work once you receive a message.

What this comes down to is scaling your consumers. When looking at NodeJS and RabbitMQ, you need to monitor message throughput to determine when to spawn new instances of your application. If you have a consistently growing queue and it never drains (has all messages processed), then you need to spawn a new instance of your consumers.

A couple of additional notes:

I highly recommend using https://github.com/arobson/rabbot for your NodeJS / RabbitMQ needs. I've worked with a lot of the "simple" NodeJS libraries for RabbitMQ and they all have limitations that I found to be unacceptable. Rabbot has a better abstraction layer to make working with RabbitMQ easier, while still giving you all the flexibility you need.

You may also want to check out my RabbitMQ and NodeJS course, here: https://sub.watchmecode.net/guides/microservices-with-rabbitmq/ - screencasts, ebooks and interviews with industry experts to help get you up to speed with RabbitMQ and NodeJS quickly. Note, though: the screecasts use Wascally as the main library - which is the predecessor to Rabbot (it got renamed to Rabbot when they made some big changes to the internals of it).

Derick Bailey
  • 72,004
  • 22
  • 206
  • 219