7

I have several workers that are being run using Sidekiq and scheduled using Sidetiq. I'm looking for advice on the best way to wait for all workers to complete before executing a callback, similar to Sidekiq-Pro's batching functionality. Any advice on possible options would be greatly appreciated!

AvocadoRivalry
  • 411
  • 1
  • 7
  • 16
  • Unfortunately the best way I have found is to "know" how many jobs are in a batch, and then keep a count when they complete (or error). – vereecjw Apr 29 '15 at 20:27
  • There's a reason I charge money for that feature: it's extremely useful and at the same time extremely difficult to build. I assume you have no budget? – Mike Perham Apr 29 '15 at 21:39
  • It's a step that I think we will ultimately be working toward, but until our ETL process is more fleshed out and we are reading in more data sources / using more workers, it isn't 100% necessary at this stage. It is certainly something on our radar, but at this point, I'm just looking for a temporary solution. – AvocadoRivalry Apr 29 '15 at 21:47

3 Answers3

5

You can write a method:

def wait_for_sidekiq
  sleep(1) until Sidekiq::Workers.new.size == 0 && Sidekiq::Queue.new.size == 0
end

I would also suggest that you make sure that the jobs got queued in the first place:

def wait_for_queuing
  sleep(1) until Sidekiq::Queue.new.size > 0 || Sidekiq::Workers.new.size > 0
end

This is required, because sometimes the first method might get executed a few miliseconds before queuing the jobs, so that it won't wait at all.

Alexander Popov
  • 23,073
  • 19
  • 91
  • 130
3

For simple use cases you can poll stats to get a count of pending jobs for each queue.

So if you know that all your jobs go to a specific queue, and there are no other jobs going in that queue from elsewhere this would suffice. For more advanced/complex use cases you can refer the API source

Another simple solution is to use a Redis based counter (since you are already using Redis) and atomically decrement it from each job, and publish an event once the count reaches zero.

lorefnon
  • 12,875
  • 6
  • 61
  • 93
1

You should see this Sidekiq-batch gem, i already try it and it works just like the Sidekiq's Pro Batch, and it's free

Hope this could help

Esgi Dendyanri
  • 497
  • 1
  • 8
  • 11