I am splitting my redis and resque workers out to a new machine. Previously they all ran on one machine - successfully.
I us cap for deploying and after a successful deploy I get this in my rails log when I try to queue up a resque job:
==> shared/log/production.log <==
I, [2016-05-28T05:43:03.924222 #5769] INFO -- : Started GET "/photos/24803/rotate/180" for 127.0.0.1 at 2016-05-28 05:43:03 +0000
I, [2016-05-28T05:43:04.080861 #5769] INFO -- : Processing by PhotosController#rotate as HTML
I, [2016-05-28T05:43:04.081274 #5769] INFO -- : Parameters: {"id"=>"24803", "degrees"=>"180"}
D, [2016-05-28T05:43:04.183430 #5769] DEBUG -- : Photo Load (1.4ms) SELECT `photos`.* FROM `photos` WHERE `photos`.`id` = 24803 LIMIT 1
I, [2016-05-28T05:43:04.250844 #5769] INFO -- : Completed 500 Internal Server Error in 169ms (ActiveRecord: 22.1ms)
F, [2016-05-28T05:43:04.256268 #5769] FATAL -- :
Redis::CannotConnectError (Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)):
app/models/photo.rb:109:in `rotate'
app/controllers/photos_controller.rb:106:in `rotate'
So I'm thinking that my app server does not get that it should go to the "backend server" for this stuff.
My configuration:
I have the app server running op 192.168.2.102 - everything is installed there except for redis. Redis is installed on 192.168.2.103
config/deploy.rb:
server '192.168.2.102', port: 22, roles: [:web, :app], primary: true
server '192.168.2.103', port: 22, roles: [:db, :resque_worker, :resque_scheduler]
set :repo_url, 'xxx'
set :application, 'xxx'
set :user, 'deploy'
set :puma_threads, [4, 16]
set :puma_workers, 0
set :workers, { "import" => 1, "utility" => 1 }
set :resque_environment_task, true
# Don't change these unless you know what you're doing
set :pty, true
set :use_sudo, false
set :stage, :production
set :deploy_via, :remote_cache
set :deploy_to, "/home/#{fetch(:user)}/apps/#{fetch(:application)}"
set :puma_bind, "unix://#{shared_path}/tmp/sockets/#{fetch(:application)}-puma.sock"
set :puma_state, "#{shared_path}/tmp/pids/puma.state"
set :puma_pid, "#{shared_path}/tmp/pids/puma.pid"
set :puma_access_log, "#{release_path}/log/puma.error.log"
set :puma_error_log, "#{release_path}/log/puma.access.log"
set :ssh_options, { forward_agent: true, user: fetch(:user) }
set :puma_preload_app, true
set :puma_worker_timeout, nil
set :puma_init_active_record, true # Change to false when not using ActiveRecord
## Defaults:
set :scm, :git
set :branch, :master
# set :format, :pretty
# set :log_level, :debug
# set :keep_releases, 5
## Linked Files & Directories (Default None):
#set :linked_files, %w{db/production.sqlite3}
set :linked_dirs, %w{ log tmp/pids tmp/cache tmp/sockets public/system }
#set :bundle_binstubs, nil
namespace :puma do
desc 'Create Directories for Puma Pids and Socket'
task :make_dirs do
on roles(:app) do
execute "mkdir #{shared_path}/tmp/sockets -p"
execute "mkdir #{shared_path}/tmp/pids -p"
end
end
before :start, :make_dirs
end
namespace :deploy do
desc "Make sure local git is in sync with remote."
task :check_revision do
on roles(:app) do
unless `git rev-parse HEAD` == `git rev-parse origin/master`
puts "WARNING: HEAD is not the same as origin/master"
puts "Run `git push` to sync changes."
exit
end
end
end
desc 'Initial Deploy'
task :initial do
on roles(:app) do
before 'deploy:restart', 'puma:start'
invoke 'deploy'
end
end
desc 'Restart application'
task :restart do
on roles(:app), in: :sequence, wait: 5 do
invoke 'puma:restart'
end
end
before :starting, :check_revision
after :finishing, :compile_assets
after :finishing, :cleanup
after :finishing, :restart
end
after "deploy:restart", "resque:restart"
# ps aux | grep puma # Get puma pid
# kill -s SIGUSR2 pid # Restart puma
# kill -s SIGTERM pid # Stop puma
config/resque.yml:
development: localhost:6379
test: localhost:6379
production: 192.168.2.103:6379
config/initializers/resque.rb
rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
rails_env = ENV['RAILS_ENV'] || 'development'
resque_config = YAML.load_file(rails_root + '/config/resque.yml')
Resque.redis = resque_config[rails_env]
Resque.logger = MonoLogger.new(File.open("#{Rails.root}/log/resque.log", "w+"))
Resque.logger.formatter = Resque::QuietFormatter.new
config/initializers/redis.rb:
$redis = Redis.new(:host => ENV["REDIS_HOST"], :port => ENV["REDIS_PORT"])
I'not sure whether I need the last file...
If you're thinking that it's my connectio setup that's wrong then think no more (about that..). Firstly Resque is not even trying to connect to the right redis. secondly, when I do this:
...on 192.168.2.103:
deploy@raspberrypi:~/apps/phototank $ netstat -nlpt | grep 6379
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN -
tcp6 0 0 :::6379 :::* LISTEN -
...on 192.168.2.102:
deploy@raspberrypi:~/apps/phototank $ redis-cli -h 192.168.2.103 ping
PONG
____EDIT____
If I run
RAILS_ENV=development rails s
on my development machine everything works perfect...what the hell?!??
