8

I am using Sidekiq in my rails app to queue up 50k+ jobs at a time. Our pool size is set to 9.

The jobs are all related and do the same thing. We have another model that has a counter on it. During each job, we check to see if that model has a column with value above 200. If it is above 200, we create another instance of that model with value = 0 and continue the jobs. However, since we have 9 jobs running at a time, all 9 jobs read the value of that column to be greater than 200 at the same time and all create new instances, which isn't right.

What's the best way to solve for this issue? We basically want all jobs to read from the most up-to-date value.

Matthew Berman
  • 8,481
  • 10
  • 49
  • 98
  • So, as I undrestand your jobs do some staff, AND each time check value and create new model? Maybe you can create 1 separate job for this staff? 8 jobs just do their work, and one job only check this value and create new instance of needed? – denys281 Jun 06 '15 at 22:52
  • Each job checks a value in the DB and only once in a while finds that they need to create a new model. However, when it does happen, all jobs currently running find they need to create a new modal at once, and create 8-10 of them. It should only create 1. – Matthew Berman Jun 08 '15 at 15:44

3 Answers3

1

I can't post any specific code because it will depend heavily on your database type and settings, but you should try database locking.

Worker when reading table should lock it until it finishes with creating new record with value 0. You should lock table for read so other workers will need to wait until this one worker finish. It is also possible to lock separate rows, but I don't know if it will work in your case.

Michał Młoźniak
  • 5,466
  • 2
  • 22
  • 38
1

Suppose your model is called Counter.

First, find/create an appropriate Counter:

counter = Counter.where('count < 200').first_or_create ...

(Note: This is not atomic! If you cannot have more than 1 active counter then see:
Race conditions in Rails first_or_create and
How do I avoid a race condition in my Rails app?)

Next, try to increment it atomically in the DB:

success = 
  Counter.where(id: counter.id)
         .where('count < 200')
         .update_all('count = count + 1')

If that worked, success == 1 otherwise success == 0. If it worked, use the counter, otherwise try again.

Community
  • 1
  • 1
Mark Bolusmjak
  • 23,606
  • 10
  • 74
  • 129
1

Redis is better suited for this type of operation and you already have easy access to it via Sidekiq's Redis connection.

value = Sidekiq.redis { |c| c.incr("my-counter") }
if value % 200 == 0
  # create new instance
end

Maybe something like that would work for you.

Mike Perham
  • 21,300
  • 6
  • 59
  • 61
  • This seems like a good, simple solution. However, after one of the jobs reads a value > 200, it needs to ping a 3rd party API and get a response. During that time, how do I make the other jobs wait until that is complete? – Matthew Berman Jun 20 '15 at 19:44
  • Sidekiq jobs are not designed to coordinate. There's no simple solution to your question. – Mike Perham Jun 20 '15 at 20:49
  • Hey @mikeperham - how about this solution: http://stackoverflow.com/questions/10750626/transactions-and-watch-statement-in-redis – Matthew Berman Jun 22 '15 at 16:44