0

First of all, this is not a "what's the best queue manager..." sort of question.

I currently use Gearman for queue management. It's fast, it's reliable, but I do miss some features and I would like to understand better options according to my needs. I use to work with Delayed_job and Resque in Ruby, but even with those I didn't found all the requirements below:

  1. I don't need a blazing fast server (we deal with ~ 3k tasks/jobs per day)
  2. I need it to be able to reenqueue a task
  3. Store current tasks in a persistent way (not memory only)
    • It should be able to replicate data among servers (in order to maintain High Availability in case of server crash)
  4. List item
    • And to be able to have a timeout that will, automatically reenqueue
  5. Be able to delay a task
    • schedule to 5 minutes from now
    • or schedule to next Sunday
  6. Be able to manage the Queue
    • How many jobs per queue?
    • What are the jobs about?
    • Can I query the Queue?

Other thoughts:

  • Being able to monitor the server status through Zabbix would be nice.

I had a great first impression about Beanstalkd. I do miss, only, data replication among servers. I also had a nice impression regarding ActiveMQ Apollo, but I didn't see how to schedule with delay.

Any thoughts?

otaviofcs
  • 785
  • 2
  • 6
  • 16
  • We use RabbitMQ for this purpose with this plugin: https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/ which allows you to queue messages with a `x-delay` value in milliseconds. – Christopher Reid Oct 17 '18 at 18:54

1 Answers1

1

Beanstalkd does most of that - and you can save the binary log. It's just local though. monitoring is easy, just connect, sends a 'STATS' call, and read the results, optionally going back to get a count from known tubes as well.

There are two other potential - remotely hosted - solutions that have high availability claims. Amazon SQS, and IronMQ. One advantage of Iron (though I've not personally used it) is that it also has an almost 100% Beanstalkd-compatible interface, just more 'enterprise' for the backend.

Alister Bulman
  • 34,482
  • 9
  • 71
  • 110
  • Thanks Alister. For now, I bet in Beanstalkd. But I do miss sync among servers. Let me check IronMQ to see how it handle data replication and I let you know. – otaviofcs Jan 08 '14 at 14:51
  • Hi Alister, as I could see IronMQ is pretty nice but it just work as a service. That for us is an issue. I would use Amazon SQS if I could use it in the cloud right now. – otaviofcs Jan 08 '14 at 17:52