How to get Sidekiq workers running on Heroku

Question

I've set up Sidekiq with my Rails project. It's running on Heroku with Unicorn. I've gone through all the configuration steps including setting the proper REDISTOGO_URL (as this question references), I've added the following to my after_fork in unicorn.rb:

after_fork do |server,worker|
    if defined?(ActiveRecord::Base)
        ActiveRecord::Base.establish_connection
        Rails.logger.info('Connected to ActiveRecord')
    end

    Sidekiq.configure_client do |config|
        config.redis = { :size => 1 }
    end
end

My Procfile is as follows:

web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec sidekiq

Right now I call my worker to perform_async and it adds the task to the queue. In fact in my Sidekiq web interface it says there are 7 items in the queue and it has all of the data there. Yet there are no workers processing the queue and for the life of me, I can't figure out why. If I run

heroku ps

I get the following output:

=== web: `bundle exec unicorn -p $PORT -c ./config/unicorn.rb`
web.1: up 2012/12/09 08:04:24 (~ 9m ago)

=== worker: `bundle exec sidekiq`
worker.1: up 2012/12/09 08:04:08 (~ 10m ago)

Anybody have any idea what's going on here?

Update

Here's the code for my worker class. Yes, I'm aware that the Oj gem has some issues potentially with sidekiq, but figured I'd give it a shot first. I'm not getting any error messages at this point (the workers don't even run).

require 'addressable/uri'
class DatasiftInteractionsWorker
include Sidekiq::Worker
sidekiq_options queue: "tweets"

def perform( stream_id , interactions )
    interactions = Oj.load(interactions)
    interactions.each{ |interaction|
        if interaction['interaction']['type'] == 'twitter'
            url = interaction['links']['normalized_url'] unless interaction['links']['normalized_url'][0].nil?
            url = interaction['links']['url'] if interaction['links']['normalized_url'][0].nil?
            begin
                puts interaction['links'] if url[0].nil?
                next if url[0].nil?
                host = Addressable::URI.parse(url[0]).host
                host = host.gsub(/^www\.(.*)$/,'\1')
                date_str = Time.now.strftime('%Y%m%d')
                REDIS.pipelined do
                    # Add domain to Redis domains Set
                    REDIS.sadd date_str , host
                    # INCR Redis host
                    REDIS.incr( host + date_str )
                end
            rescue
                puts "ERROR: Could not store the following links: " + interaction['links'].to_s
            end
        end
    }
end
end

Do you see anything in your heroku logs with [worker.1] ? something like `heroku logs -t | grep worker.1` (and then perform_async) — Jesse Wolgamott, Dec 09 '12 at 17:43
Assuming you are using the free plan, then this means that you have only access to one worker. — Benjamin Tan Wei Hao, Dec 09 '12 at 17:45
@JesseWolgamott Yes, it's up: 2012-12-09T16:04:08+00:00 heroku[worker.1]: Process exited with status 0 2012-12-09T16:04:08+00:00 heroku[worker.1]: State changed from starting to up — Nick ONeill, Dec 09 '12 at 18:01
@JesseWolgamott after running perform_async nothing happens ... It just adds to the queue but the worker never runs — Nick ONeill, Dec 09 '12 at 18:04
@NickONeill ok --- would you add the code in your worker class? — Jesse Wolgamott, Dec 09 '12 at 18:46
@JesseWolgamott in going through this process, I think I just figured it out ... since I was using a custom queue name it wasn't loading items in that queue — Nick ONeill, Dec 09 '12 at 19:19
thanks @NickONeill, looks like i have the same exact problem. is there a way to pick up items from that specific queue? — Ben Wheeler, May 05 '14 at 22:22

score 6 · Answer 1 · answered Jul 22 '13 at 18:13

6

My preference is to create a /config/sidekiq.yml file and then use worker: bundle exec sidekiq -C config/sidekiq.yml in your Procfile.

Here's an example sidekiq.yml file

answered Jul 22 '13 at 18:13

Billy Coover

3,827
5
36
50

score 4 · Answer 2 · answered Dec 10 '12 at 00:17

Figured out that if you're using a custom queue, you need to make Sidekiq aware of that queue in the Procfile, as follows:

worker: bundle exec sidekiq -q tweets, 1 -q default

I'm not quite sure why this is the case since sidekiq is aware of all queues. I'll post this as an issue on the Sidekiq project.

How to get Sidekiq workers running on Heroku

2 Answers2