Sidekiq not processing queue

Question

What possible reasons can Sidekiq prevent from processing jobs in the queue? The queue is full. The log file sidekiq.log indicates no activity at all. Thus the queue is full but the log is empty, and Sidekiq does not seem to process items. There seem to no worker processing jobs. Restarting Redis or flush it with FLUSHALL or FLUSHDB has no effect. Sidekiq has been started with

bundle exec sidekiq -L log/sidekiq.log

and produces the following log file:

2013-05-30..Booting Sidekiq 2.12.0 using redis://localhost:6379/0 with options {}
2013-05-30..Running in ruby 1.9.3p374 (2013-01-15 revision 38858) [i686-linux]
2013-05-30..See LICENSE and the LGPL-3.0 for licensing details.
2013-05-30..Starting processing, hit Ctrl-C to stop

How can you find out what went wrong? Are there any hidden log files?

0x4a6f4672 · Accepted Answer · 2013-05-30T13:30:45.347

161

The reason was in our case: Sidekiq may look for the wrong queue. By default Sidekiq uses a queue named "default". We used two different queue names, and defined them in config/sidekiq.yml

# configuration file for Sidekiq
:queues:
  - queue_name_1
  - queue_name_2

The problem is that this config file is not automatically loaded by default in your development environment (unlike database.yml or thinking_sphinx.yml for instance) by a simple bundle exec sidekiq command. Thus we wrote our jobs in two certain queues, and Sidekiq was waiting for jobs in a third queue (the default one). You have to pass the path to the config file as a parameter through the -Cor --config option:

bundle exec sidekiq -C ./config/sidekiq.yml

or you can pass the queue names directly (no spaces allowed here after the comma):

bundle exec sidekiq -q queue_name_1,queue_name_2

To find the problem out it is helpful to pass the option -v or --verbose at the command line, too, or to use :verbose: true in the sidekiq.yml file. Everything which is defined in a config file is of course useless if the config file is not loaded.. Therefore make sure you are using the right config file first.

edited May 30 '13 at 13:30

answered May 30 '13 at 12:53

0x4a6f4672

27,297
17
103
140

4

The issue for me was that my worker had `queue_as` set to a queue that wasn't in my `sidekiq.yml` file. – stephen.hanson Apr 21 '16 at 14:03
3

> or you can pass the queue names directly (no spaces allowed here after the comma): Are you sure this is fine? Shoulden't be `sidekiq -q queue_name_1 -q queue_name_2` – Wojtek Kosak Główczewski Jun 09 '16 at 07:27
the complete command should be like this `bundle exec sidekiq -d -L log/sidekiq.log -C config/sidekiq.yml -e production` along with this you should add this to your sidekiq.rb `Sidekiq.configure_server do |config| config.redis = { url: 'redis://127.0.0.1:6379' } end Sidekiq.configure_client do |config| config.redis = { url: 'redis://127.0.0.1:6379' } end` – Lalit Yadav Jun 26 '16 at 17:59
8

I don't know if it was true for the 2013, but in 2017 the sidekiq **do load** `/config/sidekiq.yml` automatically. – Aleks Feb 23 '17 at 09:33
1

As Wojtek pointed out, the command line flag `-q queue_name_1,queue_name_2` is not correct. You need to use a dedicated `-q` flag for each queue (cf. [docs](https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues)). Use commas followed by an integer to indicate queue weight. (I had attempted to correct the answer to this effect, but the edit was rejected.) – bovender Aug 11 '17 at 07:32

Edgar Ortega · Answer 2 · 2019-01-18T17:53:58.293

If you have a config/sidekiq.yml check that all the queues are defined there, check this sample file: https://github.com/mperham/sidekiq/blob/master/examples/config.yml

If you are passing queue names in the command line or Procfile, something similar to

bin/sidekiq -q queue1 -q queue2
bundle exec sidekiq -q queue1 -q queue2

check that all your queues are defined there.

In case you are not sure about the names of your queues, you can figure it out with the following script:

require "sidekiq/api"
stats = Sidekiq::Stats.new
stats.queues
# {"production_mailers"=>25, "production_default"=>1}

Then, you can do things with the queues:

queue = Sidekiq::Queue.new("production_mailers")
queue.count
queue.clear

khiav reoy · Answer 3 · 2020-10-07T09:51:15.883

It took me hours to find out that I had set config.active_job.queue_name_prefix = "xxxxx_#{Rails.env}". The queue names in the settings look the same, but sidekiq looks for the queue with prefix.

Wrong setting

app/jobs/my_job.rb

class MyJob < ApplicationJob
  queue_as :default
end

config/sidekiq.yml

:queues:
  - default

Correct setting

app/jobs/my_job.rb

class MyJob < ApplicationJob
  queue_as :default
end

config/sidekiq.yml

:queues:
  - xxxxx_development_default
  - xxxxx_production_default

score 5 · Answer 4 · answered Jan 17 '19 at 14:33

My problem was I had a configure_server but not configure_client in my initialiser, you must have both:

Sidekiq.configure_server do |config|
  config.redis = { url: ENV.fetch('SIDEKIQ_REDIS_URL', 'redis://127.0.0.1:6379/1') }
end

Sidekiq.configure_client do |config|
  config.redis = { url: ENV.fetch('SIDEKIQ_REDIS_URL', 'redis://127.0.0.1:6379/1') }
end

score 4 · Answer 5 · answered Nov 09 '18 at 19:52

In my case, sidekiq was fine in development, but stuck in staging. It was human error on the capistrano's deploy configuration. I set the path for sidekiq.yml incorrectly in the Capfile (shared instead of current).

It failed silently:

# Capfile

# WRONG:
set :sidekiq_config, -> { File.join(shared_path, 'config', 'sidekiq.yml') }
                                    ^^^^^^^^^^^
# RIGHT:
set :sidekiq_config, -> { File.join(current_path, 'config', 'sidekiq.yml') }

score 3 · Answer 6 · edited Mar 28 '22 at 17:47

3

flushing redis worked for me.

WARNING: THIS WILL REMOVE ALL DATA IN YOUR REDIS DATABASE.

redis-cli flushall

edited Mar 28 '22 at 17:47

taylorthurlow

2,953
3
28
42

answered Apr 30 '21 at 16:43

Macpeters

118
1
8

5

Warning! if you don't know what this means beforehand take a note: "Delete all the keys of all the existing databases, not just the currently selected one" (https://redis.io/commands/FLUSHALL) – AndreDurao Aug 23 '21 at 21:02

score 1 · Answer 7 · answered Apr 27 '16 at 12:57

I was banging my head against a brick wall on this for a while, my issue was that sidekiq required a newer version of redis-server. I ran "bundle exec sidekiq" and that revealed the error. Once I updated to a newer version of redis-server it was fine.

score 1 · Answer 8 · answered Mar 04 '17 at 19:23

1

I just had this issue. Turns out I had made a syntax error in my sidekiq.yml

answered Mar 04 '17 at 19:23

danini

365
5
9

score 1 · Answer 9 · answered Mar 16 '22 at 20:32

Spent at least two hours on this as well because queues and configuration and web UI were all fine ... the jobs were just not processed. My issue was that the sidekiq-server was not running in my docker-compose setup even though it should have been started in the command-section here:

 sidekiq:
    depends_on:
      - 'proddb'
      - 'redis'
    build: rails-app
    --> command: bundle exec sidekiq --environment ${RAILS_ENV} -C config/sidekiq.yml
    volumes:
      - './rails-app:/project'
      - '/project/tmp' # don't mount tmp directory
    environment:
      - REDIS_URL_SIDEKIQ=${REDIS_URL_SIDEKIQ}
    networks:
      - backend

score 0 · Answer 10 · answered Aug 11 '20 at 07:12

0

My problem was I did not config my initializers/sidekiq.rb properly but even with the correct config, sidekiq was still not running enqueued jobs. I had to run spring stop on top of that and restarted everything and it solved my issue.

answered Aug 11 '20 at 07:12

kimprap

121
7

sam-6174 · Answer 11 · 2021-04-07T00:36:52.360

I encountered a similar problem wherein the logs would show entries such as INFO Rails : queueing TestWorker (TestWorker). However, the jobs would never get processed, and none of the answers in this question solved the issue.

The tl;dr to my solution is that Sidekiq's Testing Client was getting unexpectedly triggered.

I eventually deduced that there is some "magic" going on underneath the surface that makes it difficult to discretely determine where/when/how the above testing trigger was getting configured, based on the following anecdote...

Running bundle exec sidekiq -C config/sidekiq.yml -e development had the result that Sidekiq::Testing.fake? == true

However, running bundle exec sidekiq -C config/sidekiq.yml -e development_2 had the result that Sidekiq::Testing.fake? == false

^ The only difference between these 2 commands is that I renamed the development environment in sidekiq.yml to development_2, i.e. the same/equivalent environment was running with both commands (at least, presumably it would be the same environment if it wasn't for this inane "magic" under the hood).

I updated sidekiq.rb to explicitly toggle Sidekiq::Testing via the following:

sidekiq_testing_fake = false  # set this using env var, etc.
if sidekiq_testing_fake
  Sidekiq::Testing.fake!
elsif Sidekiq.constants.include?(:Testing)
  Sidekiq::Testing.disable!
end

score 0 · Answer 12 · answered Jan 15 '22 at 14:23

0

My issue was that I had both a redis-server running and Redis.app's redis-server running, I killed the redis-server (and kept the Redis.app one)

answered Jan 15 '22 at 14:23

Dorian

7,749
4
38
57

score 0 · Answer 13 · answered Jul 12 '23 at 19:42

0

This appeared to work, but jobs sat in the queue:

bundle exec sidekiq -q queue_name,1 -c 2

This fixed things for me (I removed the weight for the queue):

bundle exec sidekiq -q queue_name -c 2

¯\_(ツ)_/¯

answered Jul 12 '23 at 19:42

hackerly

11
2

Sidekiq not processing queue

13 Answers13

Wrong setting

Correct setting

Linked