environment variables with Redis

Question

I have a Rails app that sends email notifications using Gmail (it's currently only in development mode). I set the username and password with environment variables like this

GMAIL_USERNAME='me@gmail.com' GMAIL_PASSWORD='mygreatpassword' rails s

and everything works fine. However, after I added sidekiq/redis in order to send emails as a background job, I'm getting an authentication error when it tries to send the email.

2013-04-24T19:26:45Z 887 TID-ovdxl6qxs WARN: {"retry"=>true, "queue"=>"default", "timeout"=>30, "class"=>"Sidekiq::Extensions::DelayedMailer", "args"=>["---\n- !ruby/class 'Notifier'\n- :answer_updated\n- - !ruby/object:Answer\n    attributes:\n      id: 32\n      content: Billy\n      accepted: \n      user_id: 37\n      question_id: 38\n      created_at: 2013-04-24 19:26:44.948753000 Z\n      updated_at: 2013-04-24 19:26:44.948753000 Z\n  - !ruby/object:User\n    attributes:\n      id: 35\n      email: emailaddress@gmail.com\n      encrypted_password: $2a$10$5XLIJ6F1KNPOjbTUX3fqROE0oDHgf/WnSGE4OhCm7g.pzN1bzLS0q\n      reset_password_token: \n      reset_password_sent_at: \n      remember_created_at: 2013-04-19 19:28:52.926838000 Z\n      sign_in_count: 9\n      current_sign_in_at: 2013-04-24 19:17:31.694245000 Z\n      last_sign_in_at: 2013-04-22 18:24:29.946303000 Z\n      current_sign_in_ip: 127.0.0.1\n      last_sign_in_ip: 127.0.0.1\n      created_at: 2013-04-18 19:28:09.569895000 Z\n      updated_at: 2013-04-24 19:17:31.695692000 Z\n      name: emailaddress@gmail.com\n      lawyer: \n      student: \n"], "jid"=>"4c9cb74b27080c4df581715c", "error_message"=>"530-5.5.1 Authentication Required. Learn more at\n", "error_class"=>"Net::SMTPAuthenticationError", "failed_at"=>2013-04-24 19:26:45 UTC, "retry_count"=>0}
2013-04-24T19:26:45Z 887 TID-ovdxl6qxs WARN: 530-5.5.1 Authentication Required. Learn more at

After adding sidekiq, I started the rails server the same way, with the GMAIL_USERNAME and GMAIL_PASSWORD environment variables preceding the 'rails s' command. I also started redis this way

redis-server /usr/local/etc/redis.conf

and sidekiq like this

bundle exec sidekiq

Then, after adding redis, I used sidekiq's 'delay' method instead of 'deliver' to send the message. However, as noted above, I'm getting the authentication error message.

    def after_create(answer)
      answer.question.watchers.each do |user|
      Notifier.delay.answer_updated(answer, user)
      # Notifier.answer_updated(answer, user).deliver  
    end

Update:

These are my config action mailer settings

config.action_mailer.smtp_settings = {
  address: "smtp.gmail.com",
  port: 587,
  domain: "localhost:3000",
  authentication: "plain",
  enable_starttls_auto: true,
  user_name: ENV["GMAIL_USERNAME"],
  password: ENV["GMAIL_PASSWORD"]
}

This is what it says after "learn more at"

2013-04-25T00:11:30Z 6904 TID-owuofcbno WARN: /Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/1.9.1/net/smtp.rb:948:in `check_response'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/1.9.1/net/smtp.rb:917:in `getok'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/1.9.1/net/smtp.rb:832:in `mailfrom'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/1.9.1/net/smtp.rb:659:in `send_message'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/mail-2.4.4/lib/mail/network/delivery_methods/smtp.rb:145:in `block in deliver!'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/1.9.1/net/smtp.rb:520:in `start'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/mail-2.4.4/lib/mail/network/delivery_methods/smtp.rb:144:in `deliver!'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/mail-2.4.4/lib/mail/message.rb:2034:in `do_delivery'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/mail-2.4.4/lib/mail/message.rb:229:in `block in deliver'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/actionmailer-3.2.12/lib/action_mailer/base.rb:415:in `block in deliver_mail'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/activesupport-3.2.12/lib/active_support/notifications.rb:123:in `block in instrument'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/activesupport-3.2.12/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/activesupport-3.2.12/lib/active_support/notifications.rb:123:in `instrument'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/actionmailer-3.2.12/lib/action_mailer/base.rb:413:in `deliver_mail'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/mail-2.4.4/lib/mail/message.rb:229:in `deliver'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/extensions/action_mailer.rb:23:in `perform'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/processor.rb:49:in `block (3 levels) in process'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/middleware/chain.rb:109:in `call'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/middleware/chain.rb:109:in `block in invoke'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/middleware/server/timeout.rb:11:in `block in call'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/middleware/server/timeout.rb:10:in `call'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/middleware/chain.rb:111:in `block in invoke'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/middleware/chain.rb:111:in `block in invoke'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/middleware/server/retry_jobs.rb:50:in `call'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/middleware/chain.rb:111:in `block in invoke'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/logging.rb:22:in `with_context'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/middleware/server/logging.rb:7:in `call'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/middleware/chain.rb:111:in `block in invoke'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/middleware/chain.rb:114:in `call'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/middleware/chain.rb:114:in `invoke'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/processor.rb:48:in `block (2 levels) in process'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/processor.rb:87:in `stats'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sidekiq-2.10.1/lib/sidekiq/processor.rb:47:in `block in process'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/celluloid-0.12.4/lib/celluloid/calls.rb:23:in `call'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/celluloid-0.12.4/lib/celluloid/calls.rb:23:in `public_send'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/celluloid-0.12.4/lib/celluloid/calls.rb:23:in `dispatch'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/celluloid-0.12.4/lib/celluloid/future.rb:18:in `block in initialize'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/celluloid-0.12.4/lib/celluloid/internal_pool.rb:48:in `call'
/Users/me/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/celluloid-0.12.4/lib/celluloid/internal_pool.rb:48:in `block in create'

also, can you add your `config.action_mailer.smtp_settings` to see how you are setting your passwords there? — jstim, Apr 24 '13 at 22:24
@jstim I updated the OP with the information you asked for. Thanks for your help. — Tomoko Yamaguchi, Apr 25 '13 at 00:14
@TomokoYamaguchi weird are u sure it working without `sidekiq` also check if gmail not throwing any error If you sending repeated mail in small time I observer the gmail wont allow you do that it either throw and error or present a captcha to avoid it check [this](http://stackoverflow.com/questions/1607828/how-to-use-gmail-as-a-free-smtp-server-and-overcome-captcha) hope this help — Viren, Apr 25 '13 at 16:17
@Viren it's definitely working fine without sidekiq/redis. I've tried it multiple times. I'm only getting authentication errors when I use sidekiq/redis. I also haven't reached any of the daily gmail limits. Tested this over multiple days. — Tomoko Yamaguchi, Apr 25 '13 at 18:47
@TomokoYamaguchi Ok wait a minute you are starting the sidekiq without any environment options like this `bundle exec sidekiq` perhaps you need to do something like this `GMAIL_USERNAME='me@gmail.com' 'GMAIL_PASSWORD='mygreatpassword' bundle exec sidekiq` if you dont want to pass like this either define them as constant in application or defined them in bashrc then it would be available as `ENV['GMAIL_USERNAME']` and `ENV['GMAIL_PASSWORD']` — Viren, Apr 26 '13 at 06:25

score 10 · Accepted Answer · answered Apr 27 '13 at 08:57

Sidekiq starts it's own rails environment and is missing the environment variables for configuring ActionMailer, this is why you get the exception when using sidekiq, but not when sending within the rails server process itself.

Basically you just have to start the sidekiq worker using the same variables like the server:

GMAIL_USERNAME='me@gmail.com' GMAIL_PASSWORD='mygreatpwd' bundle exec sidekiq

Though this might make work okay, consider to use something like foreman to handle this (especially in development).

score 4 · Answer 2 · answered Apr 26 '13 at 02:35

My guess is that the environment variables you are setting in your console are only being applied to your web server's process (which is started by rails s). So when sidekiq looks at the mailer config it gets back nil from ENV["GMAIL_USERNAME"] and ENV["GMAIL_PASSWORD"]. You need to put those environment variables where the sidekiq process can access them.

I'll lay out a few options below (#2 looks easiest), but there are already a few resources out there that have explained this in more detail (and credit goes to them for compiling these options). Check out:

http://railsapps.github.io/rails-environment-variables.html
https://devcenter.heroku.com/articles/config-vars#local-setup
- If you are deploying on heroku and want to test locally
- Just remember to put .env in .gitignore

Options

Setting environment vars in bash

I would not recommend this method, but wanted to include it as a counterpoint to the two options below.

export GMAIL_USERNAME="myemail@gmail.com"

Running this in your bash console will create a value in ~/.bashrc accessible in ENV.

This solution, however, is not application-specific (it is available anywhere on your machine), and it is buried so far away from your application you may find it difficult to debug issues or forget to delete/change this private information.

Figaro gem

The figaro gem provides you with a .gitignored YAML file that you can put your sensitive config information.

It also provides helpers for uploading these config vars to heroku, while still keeping the config out of your git repos.

Reading from a yml config file

This is pretty much what the Figaro gem is doing, but it's worth understanding.

# 1. create a config file
touch config/gmail_config.yml
# 2. add the new file to .gitignore
# 3. add code to application.rb to load yml file
config.before_configuration do
  env_file = File.join(Rails.root, 'config', 'local_env.yml')
  YAML.load(File.open(env_file)).each do |key, value|
    ENV[key.to_s] = value
  end if File.exists?(env_file)
end

If the file exits (ie you are on your local machine) it will load up the file and set the variable. If the file doesn't exist, it will not load the file and you will need to set the variable in your production environment.

Thanks a lot for the information. I'll give you an upvote for the userful info. However, I'll give the bounty to the other answer because it was a more direct response to the question. — Tomoko Yamaguchi, Apr 27 '13 at 19:48

environment variables with Redis

2 Answers2

Options

Setting environment vars in bash

Figaro gem

Reading from a yml config file