0

I am using resque to background process two types of jobs:

(1) 3rd-party API requests 
(2) DB query and insert

While the two jobs can be processed parallely, each job type in itself can only be processed in serial order. For example, DB operations need to happen in serial order but can be executed in parallel with 3rd party API requests.

I am contemplating either of the following methods for executing this :

(1) Having two queues with one queue handling only API requests and the other queue 
handling only db queries. Each queue will have its own worker.

(2) One single queue but two workers. One worker for each job.

I would like to know the difference in the two approaches and which among the two would be a better approach to take.

Ninja
  • 5,082
  • 6
  • 37
  • 59

1 Answers1

1

This choice of selecting an architecture is not straight forward, you have to keep many things in mind.

Having two queues with one queue handling only API requests and the other queue handling only db queries. Each queue will have its own worker.

Ans: You can have this architecture when you have both the queues equally busy i.e if you have one queue with more number of jobs and other empty then your one worker will be idol and there will be jobs waiting in other queue.
You should always think about full utilisation of your workers.

One single queue but two workers. One worker for each job.

Ans: This approach is what we also use in our project.
Having all jobs en-queued to one queue and have multiple workers running on it.
Your workers will always be busy no matter which type job is present.
proper worker utilisation is possible.


At last I would suggest you can use 2nd approach.

ritzz.soni
  • 335
  • 1
  • 15