28

Please suggest an equivalent of Celery in Node JS to run asynchronous tasks. I have been able to search for the following:

  1. (Later)
  2. Kue (Kue),
  3. coffee-resque (coffee-resque)
  4. cron (cron)
  5. node-celery(node celery)

I have run both manual and automated threads in background and interact with MongoDB.

node-celery is using redis DB and not Mongo DB. Is there any way I can change that?When I installed node-celery redis was installed as dependency.

I am new to celery, Please guide.Thanks.

garima
  • 5,154
  • 11
  • 46
  • 77

5 Answers5

19

Celery is basically a RabbitMQ client. There are producers (tasks), consumers (workers) and AMQP message broker which delivers messages between tasks and workers.

Knowing that will enable you to write your own celery in node.js.

enter image description here

node-celery here is a library that enables your node process to work both as a celery client (Producer/Publisher) and a celery worker (Consumer).

See https://abhishek-tiwari.com/post/amqp-rabbitmq-and-celery-a-visual-guide-for-dummies

Community
  • 1
  • 1
Vanuan
  • 31,770
  • 10
  • 98
  • 102
  • 4
    I think that you are not having into account all the things that celery solves for you. Is not just a tiny layer on top of AMQP. – Joac Mar 26 '21 at 19:41
  • @Joac +1, for 99% of users just using [celery-node](https://www.npmjs.com/package/celery-node) makes much more sense than trying to write your own from scratch as suggested in this answer. – user456584 Jun 01 '22 at 17:41
  • @user456584 celery-node didn't exist at the time of this answer, duh – Vanuan Jun 02 '22 at 23:50
  • @Vanuan Does it exist today? Allow me to rephrase: "for 99% of users just using celery-node makes much more sense than trying to write your own from scratch" – user456584 Jun 04 '22 at 00:20
7

Edit-1/2018

My recommendation is not to use Kue now, as it seems to be a stalled project, use Celery instead. It is very well supported and maintained by the community and supports large number of use cases.

Old Answer

Go for Kue, it's a wholistic solution that resembles Celery in Python word; it has the concepts of: producers/consumers, delayed tasks, task retrial, task TTL, ability to round-robin tasks across multiple consumers listening to the same queue, etc.

Probably Celery is more advanced with more features with more brokers to support and you can use celery-node if you like, but, in my opinion, I think no need to go for a hybrid solution that requires installation of python and node when you can only use only language that's sufficient in 90% of the cases (unless necessary of course).

user456584
  • 86,427
  • 15
  • 75
  • 107
securecurve
  • 5,589
  • 5
  • 45
  • 80
6

Go for Kue, it's a wholistic solution that resembles Celery in Python word; it has the concepts of: producers/consumers, delayed tasks, task retrial, task TTL, ability to round-robin tasks across multiple consumers listening to the same queue, etc.

Kue, after so much time have passed, still has the same old core issues unsolved:

  • github.com/Automattic/kue/issues/514
  • github.com/Automattic/kue/issues/130
  • github.com/Automattic/kue/issues/53

If anyone reading this don't want to rewrite Kue, don't start with it. It's good for simple tasks. But if you want to deal with a lot of them, concurrent, or task chains (when one task creates another) - stop wasting your time.

I've wasted a month trying to debug Kue and still no success. The best choice was to change Kue for Pubs/sub Messaging queue on RabbitMQ and Rabbot (another RabbitMQ wrap up).

Personally, I haven't used Celery as much to put all in for it, but as I've been searching for Celery alternative and found how someone is advising for Kue just boiled my blood in veins.

If you want to send a delayed email (as in Kue example) you can go for whatever you'd like without worrying about errors. But if you want a reliable system task/message queue, don't even start with Kue. I'd personally go with 5. node-celery(node celery)

Darvydas Šilkus
  • 329
  • 3
  • 4
  • 1
    Thanks for caring enough to post your experience here. This made me laugh and at the same time sympathise -> "boiled my blood in veins". I have felt the exact same way when certain over-hyped web applications are mentioned. I have wasted so many months battling those applications people carelessly recommend just because they are the trend. – a20 Nov 28 '17 at 17:42
  • The answer that you are quoting is 4 years old, if you take an old post and base your solution on without validation first, then, that's your mistake ... Nowadays, I would recommend people to use Celery all the way, as it is a very well maintained library. – securecurve Jan 09 '18 at 08:31
  • We learn from our mistakes. At the end of 2016, when I was searching for a solution Kue seemed to be it. And only when you dig deep enough you see that it's not what you were looking for. Shared my hard feelings for others not to choose Kue (which is still placed as second on the question). And as I see, since mid 2017 Kue have never got back up. @securecurve thanks for updating you answer ;) – Darvydas Šilkus Dec 07 '18 at 22:06
6

It is also worth mentioning https://github.com/OptimalBits/bull. It is a fast, reliable, Redis-based queue written for stability and atomicity.

Bull 4 is currently in beta and has some nice features https://github.com/taskforcesh/bullmq

jsbroks
  • 530
  • 6
  • 15
2

In our experience, Kue was unreliable, losing jobs. Granted, we were using an older version, it's probably been fixed since. That was also during the period when TJ abandoned the project and the new maintainers hadn't been chosen. We switched to beanstalkd and have been very happy. We're using https://github.com/ceejbot/fivebeans as the node interface to beanstalkd.

Bryan Larsen
  • 9,468
  • 8
  • 56
  • 46