Sidekiq deploy to multiple environments

Question

(See below for my detailed config, which is the result of Henley Chiu's answer).

I've been trying to wrap my brain around Sidekiq deploys, and I am not really getting it. I have an app with a staging environment, and a production environment, on the same server. Everything I see about sidekiq deploys basically say "just add sidekiq/capistrano to your deploy file", so I did that. And then the instructions are "here's a yml file with options" but nothing seems to be explained. Do I need namespaces? I see that in an initialize file, but that seems to be to point outside the server.

I deployed earlier, and each stage seems to boot sidekiq up with the proper environment, but they both process from the same queues. My emails from production were trying to be processed by the stage sidekiq, and failing. I stopped my stage for now, but eventually I will need to use it again. I hope I'm not being dense, I've really tried to understand this and am just having a hard time with finding a definitive "here's how it's done".

For what it's worth, here is config/sidekiq.yml (which is loaded fine during the deploy):

:concurrency: 5
:verbose: false
:pidfile: ./tmp/pids/sidekiq.pid
:logfile: ./log/sidekiq.log
:queues:
  - [carrierwave, 7]
  - [client_emails, 5]
  - [default, 3]
staging:
  :concurrency: 10
production:
  :concurrency: 25

Log files, and pids seem to be in the right spot, but the queues are just merged. Any help would be GREAT!

Also, if it matters:

Rails 3.2.11, passenger, nginx, rvm, Ubuntu 12.10, and Ruby 1.9.3

Detailed Configuration (answer):

First I set up a new redis server at port 7777 (or whatever port you please besides the default 6379). Pretty much followed the redis quickstart guide that I used the first time around.

Then I made the initilizer file; this has both the client and the server config. Both are required to make sidekiq work multistage.

Note that I am using an external YAML file for the settings. I am using SettingsLogic for this to make things easier, but you can just as easily do this yourself by including the file. By using a yaml file, we don't have to touch our environments/staging or production files.

# config/initializers/sidekiq.rb
server = Settings.redis.server
port = Settings.redis.port
db_num = Settings.redis.db_num
namespace = Settings.redis.namespace

Sidekiq.configure_server do |config|  
  config.redis = { url: "redis://#{server}:#{port}/#{db_num}", namespace: namespace  }
end

I am using passenger - the troubleshooting page of the sidekiq wiki recommends a change for the setup when using unicorn or passenger, so I added the code there for the client setup:

# config/initializers/sidekiq.rb (still)
if defined?(PhusionPassenger)
  PhusionPassenger.on_event(:starting_worker_process) do |forked|
    Sidekiq.configure_client do |config|
      config.redis = { url: "redis://#{server}:#{port}/#{db_num}", namespace: namespace }
    end if forked
  end
end

This is my Settings file (obviously values changed):

#config/settings.yml
defaults: &defaults
  redis: &redis_defaults
    server: 'localhost'
    port: 6379
    db_num: 0
    namespace: 'sidekiq_development'

development:
  <<: *defaults

test:
  <<: *defaults

staging:
  <<: *defaults
  redis:
    <<: *redis_defaults
    port: 8888
    namespace: 'sidekiq_staging'

production:
  <<: *defaults
  redis:
    <<: *redis_defaults
    port: 7777
    namespace: 'sidekiq_production'

I found that adding the namespace to the config/sidekiq.yml file didn't seem to work - sidekiq would boot on deploy using the right port, but wouldn't actually process anything. But since the wiki recommends using a namespace, I ended up just adding it to the init file.

I hope this helpful for others, because this was really hard for me to understand, having not done a lot of this kind of setup before.

score 20 · Answer 1 · answered Oct 23 '13 at 11:30

20

If all environments(development, staging and production) are on same server then use namespace. In your initializers/sidekiq.rb file,

Sidekiq.configure_server do |config|
    config.redis = { url: 'redis://localhost:6379/0', namespace: "sidekiq_app_name_#{Rails.env}" }
end

Sidekiq.configure_client do |config|
    config.redis = { url: 'redis://localhost:6379/0', namespace: "sidekiq_app_name_#{Rails.env}" }
end

answered Oct 23 '13 at 11:30

Ranjithkumar Ravi

3,352
2
20
22

How do i specify the queues in my worker when using namespace. Do i need to change the name of the queues as well? – nik-v Nov 13 '13 at 11:36
@nik-v I guess you found out by yourself, but no, you don't need to change anything else except add this initializer. – janosrusiczki May 28 '14 at 20:33
yeah had figured that out. Basically there was some other mistake I was doing because of which it didnt work. – nik-v Jun 02 '14 at 13:10
Will sidekiq access different databases for each environment? I guess not. So, separating only the namespace will not help much if there is database access needed. – cool_php Feb 09 '23 at 16:05

score 4 · Accepted Answer · answered Feb 12 '13 at 17:10

4

In your initializers/sidekiq.rb file, you specify the Redis queue all environments boot up with. For mine it is:

redisServer = "localhost"
Sidekiq.configure_server do |config|
  config.redis = { :url => 'redis://' + redisServer + ':6379/0' }
end

If you want each environment to process from separate queues, you can have specific sidekiq.rb files in the environments folder for each environment. Each with different redis servers.

answered Feb 12 '13 at 17:10

Henley

21,258
32
119
207

Ah, this makes more sense - I need another redis server. For some reason I hadn't thought of that. I will try this out, and if it fixes things I'll be back to mark this as the answer. Thanks! – d3vkit Feb 12 '13 at 19:02
I had to make some minor adjustments but your instructions got me there, so I have marked this as the answer! Thank you for taking the time to help! – d3vkit Feb 14 '13 at 05:19
@d3vkit I am also running into same problem. Just wanted to make sure. I currently have multiple environments(staging and production) running on same server. My workers also need access to ActiveRecord (different database for different environment). Can you suggest how to go about it. Can I do with 1 redis server? – nik-v Nov 13 '13 at 11:03

score 3 · Answer 3 · answered Jul 08 '14 at 10:34

In addition to the namespace, it will be good if you also separate out DBs for each Rails environment in Redis too i.e.:

env_num = Rails.env == 'staging' ? 0 : 1
Redis.new(db: env_num) # existing DB is selected if already present

Sidekiq.configure_server do |config|
  config.redis = { url: "redis://localhost:6379/#{env_num}", namespace: "app_name_#{Rails.env}" }
end

Sidekiq.configure_client do |config|
  config.redis = { url: "redis://localhost:6379/#{env_num}", namespace: "app_name_#{Rails.env}" }
end

Sidekiq deploy to multiple environments

Detailed Configuration (answer):

3 Answers3

Linked