What are good message queue options for nodejs?

Question

Looking to use a message queue in a small web app I'm building with node.js. I looked at resque but not sure that's appropriate. The goal is to push notifications to clients based on backend and other client actions with socketio. I could do this with just socketio but I thought maybe a proper message queue would make this cleaner and I wouldn't have to reinvent the wheel.

What are the options out there?

Not sure, but this seems like something node would do well by itself! — TK-421, Jan 15 '11 at 17:28
You're probably aware of this already, but there's one listed on the Modules page: https://github.com/ry/node/wiki/modules#message-queue . I guess there's always the cost of your own development time to consider. — TK-421, Jan 15 '11 at 17:40
@ TK-421 and Bjorn Tipling That's indeed something node can do itself, as long as you only have one node process. An external solution such as Redis is needed if you have different processes for different parts of your application (i.e. webserver, auth provider, notif center etc.). And of course you can then connect with non node processes as well. — Louis Chatriot, Dec 14 '12 at 09:40
Examples using Node AMQ and Rabbit MQ (Producer) https://gist.github.com/DarcInc/9641557 and (Consumer) https://gist.github.com/DarcInc/9641582 — ipaul, Mar 19 '14 at 13:27
In case you need an in-memory queue you can consider this [rxjs-based solution](https://stackoverflow.com/a/46539032/1555615) — Marinos An, Oct 31 '19 at 14:36

score 57 · Accepted Answer · edited May 23 '17 at 12:10

57

you could use redis with the lightning fast node_redis client. It even has built-in pubsub semantics.

edited May 23 '17 at 12:10

Community

1
1

answered Jan 15 '11 at 20:37

Alfred

60,935
33
147
186

8

I recommend using a good queueing module on top of redis, just like RSMQ, which seems to be pretty simple. https://www.npmjs.com/package/rsmq – Exinferis Jan 19 '15 at 11:32
if you just want the power of messaging without the technical concerns creeping in then you can try https://node-ts.github.io/bus/ – Andrew dh Apr 29 '19 at 13:21

score 14 · Answer 2 · answered Apr 15 '11 at 17:28

You could use the node STOMP client. This would let you integrate with a variety of message queues including:

ActiveMQ
RabbitMQ
HornetQ

I haven't used this library before, so I can't vouch for its quality. But STOMP is a pretty simple protocol so I suspect you can hack it into submission if necessary.

Another option is to use beanstalkd with node. beanstalkd is a very fast "task queue" written in C that is very good if you don't need the feature flexibility of the brokers listed above.

Josh Bassett · Answer 3 · 2012-03-25T03:38:52.290

12

Shameless plug: I'm working on Bokeh: a simple, scalable and blazing-fast task queue built on ZeroMQ. It supports pluggable data stores for persisting tasks, currently in-memory, Redis and Riak are supported. Check it out.

edited Mar 25 '12 at 03:38

answered Mar 22 '12 at 00:12

Josh Bassett

121
1
5

RobotEyes · Answer 4 · 2018-06-08T08:58:52.097

10

Here's a couple of recommendations I can make:

node-amqp: A RabbitMQ client that I have successfully used in combination with Socket.IO to make a real-time multi-player game and chat application amongst other things. Seems reliable enough.

zeromq.node: If you want to go down the non-brokered route this might be worth a look. More work to implement functionality but your more likely to get lower latency and higher throughput.

edited Jun 08 '18 at 08:58

answered Aug 22 '12 at 22:41

RobotEyes

4,929
6
42
57

1

+1 on using ZeroMQ. After a lot of research and time spent tinkering with beanstalkd, RabbitMQ, BeeQueue, Bull and Kue, ZeroMQ ended up being the best experience for me, especially for lightweight projects powered by workers. It's lightning fast and the documentation is top notch. It also has the added benefit of not clogging up your Redis server with a large amount of calls. – dimiguel Jun 08 '18 at 03:30
`zeromq.node` now maintained here: [zeromq.js](https://github.com/zeromq/zeromq.js) – Marinos An Oct 31 '19 at 09:58

score 9 · Answer 5 · answered Jun 16 '15 at 06:40

Take a look at node-busmq - it's a production grade, highly available and scalable message bus backed by redis.

I wrote this module for our global cloud and it's currently deployed in our production environment in several datacenters around the world. It supports named queues, peer-to-peer communication, guaranteed delivery and federation.

For more information on why we created this module you can read this blog post: All Aboard The Message Bus

score 7 · Answer 6 · answered Jun 15 '13 at 23:36

7

kue is the only message queue you would ever need

answered Jun 15 '13 at 23:36

Pono

11,298
9
53
70

33

except kue is not well maintained, has several issues and not a single test ! – vvo Nov 28 '13 at 10:36
4

Also, it's a job queue - not a message queue – HyderA Mar 08 '14 at 22:11
It has several issues and is not fit for production – Rahul Kumar Aug 31 '18 at 16:24
2

Using `bull` was more straightforward to me. With `kue` I was lost in the documentation. – Marinos An Oct 31 '19 at 09:26

score 7 · Answer 7 · answered May 30 '11 at 05:29

7

I recommend trying Kestrel, it's fast and simple as Beanstalk but supports fanout queues. Speaks memcached. It's built using Scala and used at Twitter.

answered May 30 '11 at 05:29

Eduardo Raad

169
1
11

7

It is worth noting that Kestrel is no longer in active development. – GordyD Oct 02 '14 at 08:38

score 5 · Answer 8 · answered Dec 19 '14 at 12:08

5

You might want to have a look at

Redis Simple Message Queue for Node.js

Which uses Redis and offers most features of Amazons SQS.

answered Dec 19 '14 at 12:08

Smrchy

83
1
4

1

While RSMQ is nice and has been working for me in production once, be aware that it's using Lua scripts in Redis and will not work with Redis cluster/sentinel setup – naugtur Jun 11 '17 at 11:53

AndyGrom · Answer 9 · 2014-05-23T20:51:29.427

Look at node-queue-lib. Perhaps it is enough that you. It support node.js and browsers. Has two delivery strategies: broadcast and round-robin. Only javascript.

Quick example:

var Queue = require('node-queue-lib/queue.core');

var queue = new Queue('Queue name', 'broadcast');

// subscribe on 'Queue name' messages
queue.subscribe(function (err, subscriber) {
    subscriber.on('error', function(err){
        //
    });
    subscriber.on('data', function (data, accept) {
        console.log(data);
        accept(); // accept process message
    });
});

// publish message
queue.publish('test');

score 3 · Answer 10 · answered May 27 '15 at 16:05

3

How about Azure ServiceBus? It supports nodejs.

answered May 27 '15 at 16:05

Ben

723
4
10
18

score 2 · Answer 11 · edited Jun 01 '17 at 08:01

I used KUE with socketIO like you described. I stored the socketID with the job and could then retreive it in the Job Complete.. KUE is based on redis and has good examples on github

something like this....

jobs.process('YourQueuedJob',10, function(job, done){
    doTheJob(job, done);
});


function doTheJob(job, done){
    var socket = io.sockets.sockets[job.data.socketId];
    try {
        socket.emit('news', { status : 'completed' , task : job.data.task });
    } catch(err){
        io.sockets.emit('news', { status : 'fail' , task : job.data.task , socketId: job.data.socketId});
    }
    job.complete();
}

score 1 · Answer 12 · answered Jul 29 '16 at 09:37

1

You might also want to check out ewd-qoper8: https://github.com/robtweed/ewd-qoper8

answered Jul 29 '16 at 09:37

robtweed

21
2

What are good message queue options for nodejs?

12 Answers12

Linked