1

I am new to queueing systems. I am using resque with my current app and I have three queues to monitor. I wonder what is the best way to run and manage resque jobs. The reason for the question is that I am slightly confused with so many options suggested by developers on the internet. More specifically my confusion is around these two strategies.

  • resque pool
  • resque pool with god

why do I need resque pool if I can run workers with god alone?

Sunil D.
  • 17,983
  • 6
  • 53
  • 65
Rahul
  • 442
  • 6
  • 17
  • Is your query solved? – ritzz.soni Jun 11 '16 at 08:32
  • Not really, I am still confused with the plethora of gems available to do such a basic thing. I plan to deploy my app with capistrano+Chef. Running god is added complexity which seems inevitable. But I am not really able to understand the need of resque pool. seems like too many tools to kill the mouse. There is no justification for using resque pool or otherwise. Not even an analysis of the tradeoff if any. – Rahul Jun 11 '16 at 13:24
  • Can you explain me your needs? How frequent is your worker running? – ritzz.soni Jun 13 '16 at 06:17
  • My app has three queues to start with, a mailer queue, a logger queue and a digest mailer queue. I plan to run digest mailer queue as a chrojob. – Rahul Jun 14 '16 at 13:49

2 Answers2

3

Resque was designed to handle background jobs, basically to handle jobs which stays in queue for shorter time.
It came into the picture as a replacement for Delayed Job. When your background jobs are increasing you should opt for Resque.
Now before talking about monitoring I would talk a little about how does Resque works.

  1. Resque has Redis as its backend which is a In memory database, having redis as backend makes it faster.

  2. Resque has processes which will take out the jobs and execute them, worker takes the job and fork a child which will perform the job and exits. So typically one fork per job is what Resque provide by default.

  3. Now there is a vast variety of plugins are present to enhance the functionality of Resque. For Example: Resque Pool,

Why we need it and when: People have their requirement like one worker multiple jobs per fork. People want these things so these plugins are help full.

  • Resque Pool:is a daemon for managing a pool of resque workers. With a simple config file, it manages your workers for you, starting up the appropriate number of workers for each worker type. So it makes your life easy as you can manage your workers with this.
  • God is also there as a tool for monitoring your workers basically all these are there to enhance existing Resque functionality.
  • You can also look for Resque-web it is web portal for monitoring your Resque system:

You can follow given links Which were help full to me: Redis Persistence: http://redis.io/topics/persistence

How Resque works with Redis : http://girders.org/blog/2011/10/30/how-queuing-with-resque-works/

Resque-web: https://github.com/resque/resque-web

Tutorial : http://www.sitepoint.com/simple-organized-queueing-with-resque/

Multiple redis servers : What's the Point of Multiple Redis Databases?

redis-databases

Resque-scheduler : https://github.com/resque/resque-scheduler

Resque-pool for managing workers : https://github.com/nevans/resque-pool

Resque-multi-jobs-fork : https://github.com/stulentsev/resque-multi-job-forks

Deploy in production: How to deploy resque workers in production? deploy-resque-workers-in-production?rq=1

Community
  • 1
  • 1
ritzz.soni
  • 335
  • 1
  • 15
-2

Running jobs are usually asynchronous tasks. It means you dont wait for them to finish to keep going with your request or whatever called the job.

Because jobs nature is asynchronous, it may mean that at some point you are requesting several jobs at one time. Due to CPU restrictions, these jobs will either find a busy cpu and wait or fail.

A resque pool is a way to keep them in line, performed one at a time. If they fail, they go to a retry mode sometimes, etc.

God is a daemon monitoring tool. It will monitor the jobs 'server' to allow accepting jobs, but not necessarily handle failures, or handle jobs in the order they came in, etc.

A good tool that tries to encapsulate all these into one is sidekiq. There is also a railscast about it that explains well how to work with it (even though it may be a bit old): http://railscasts.com/episodes/366-sidekiq

Marcelo Ribeiro
  • 1,718
  • 1
  • 13
  • 27
  • 1
    Thanks for answer... Is there any senario when resque-pool might be avoidable. The reason I am asking this is because I feel it's an added complexity. – Rahul Jun 11 '16 at 13:42