39

I keep getting an OpenSSL::SSL::SSLError with a message of SSL_read: cert already in hash table when sending delayed e-mails out asynchronously with ActionMailer.

We use Sidekiq to send all of our e-mails out asynchronously. When posing this question as an issue in the Sidekiq GitHub repo, I was told that Sidekiq doesn't know anything about or manage the SSL connection.

Our app is hosted on Heroku, which is running OpenSSL 0.9.8k 25 Mar 2009.

We've seen this error several times in other jobs and have found that sometimes the jobs get processed, but sometimes they don't.

Is this an OpenSSL threading problem in which multiple Sidekiq threads are attempting to use the same SSL connection? Is a there a fix to this?

Here's the stacktrace we're getting:

    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/openssl/buffering.rb:175:in `sysread_nonblock`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/openssl/buffering.rb:175:in `read_nonblock`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:153:in `rbuf_fill`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:134:in `readuntil`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:144:in `readline`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:932:in `recv_response`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:903:in `block in data`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:942:in `critical`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:896:in `data`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:663:in `block in send_message`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:852:in `rcptto_list`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:663:in `send_message`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/network/delivery_methods/smtp.rb:113:in `block in deliver!`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:521:in `start`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/network/delivery_methods/smtp.rb:112:in `deliver!`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/message.rb:2129:in `do_delivery`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/message.rb:232:in `block in deliver`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/actionmailer-4.0.3/lib/action_mailer/base.rb:456:in `block in deliver_mail`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.3/lib/active_support/notifications.rb:159:in `block in instrument`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.3/lib/active_support/notifications/instrumenter.rb:20:in `instrument`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.3/lib/active_support/notifications.rb:159:in `instrument`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/actionmailer-4.0.3/lib/action_mailer/base.rb:454:in `deliver_mail`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/message.rb:232:in `deliver`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/extensions/action_mailer.rb:20:in `perform`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:49:in `block (3 levels) in process`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:122:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:122:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm-3.7.1.182/lib/new_relic/agent/instrumentation/sidekiq.rb:30:in `block in call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm-3.7.1.182/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:339:in `perform_action_with_newrelic_trace`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm-3.7.1.182/lib/new_relic/agent/instrumentation/sidekiq.rb:21:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-failures-0.3.0/lib/sidekiq/failures/middleware.rb:10:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-unique-jobs-2.7.0/lib/sidekiq-unique-jobs/middleware/server/unique_jobs.rb:15:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/server/active_record.rb:6:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/server/retry_jobs.rb:62:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/server/logging.rb:11:in `block in call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/logging.rb:22:in `with_context`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/server/logging.rb:7:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:127:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:127:in `invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:48:in `block (2 levels) in process`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:105:in `stats`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:47:in `block in process`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:86:in `do_defer`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:37:in `process`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `public_send`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `dispatch`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:122:in `dispatch`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:322:in `block in handle_message`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:416:in `block in task`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:55:in `block in initialize`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/tasks/task_fiber.rb:13:in `block in create
Ricky Zein
  • 557
  • 1
  • 4
  • 8
  • 11
    Why is Heroku running an OpenSSL from 2009? That's terrifying. – tadman May 22 '15 at 19:08
  • 3
    don't see the error in the stack trace. is the trace complete? – Mircea Jun 13 '15 at 19:56
  • have you checked this post: http://stackoverflow.com/questions/5711190/how-to-get-rid-of-opensslsslsslerror – tokhi Jun 24 '15 at 11:50
  • 6
    I just ran `heroku run bash --remote production` and then `openssl version` and see `OpenSSL 1.0.1f 6 Jan 2014`. Have you considered updating your stack? https://devcenter.heroku.com/articles/stack – Dan Croak Nov 05 '15 at 06:27
  • 3
    Since this question has no activity for last 2 years and the OP has disappeared, can this be closed or something ? – Sam Mar 16 '16 at 14:08
  • 1
    I'm voting to close this question as off-topic because it appears to be related to an OpenSSL issue from 2009. – Andy Mikula Mar 17 '16 at 14:47
  • 2
    @AndyMikula That an issue occurred for an old version, or might be fixed by a newer version, is not by itself a valid close reason. If the solution is indeed known to be that one should update the stack, that should be added to the question as an answer. People run old stacks all the time, so this answer could still help someone. 988 views indicates that this question either does, or did, draw people to it looking for an answer. – Wayne Conrad Jun 12 '16 at 00:59
  • I'm betting that the SSL code is/was not concurrent-safe and that's where the problem is coming from. – David Ljung Madison Stellar Jun 22 '16 at 07:03
  • I’m voting to close this question because old issue of heroku stack no answer of OP – Roland Studer Jul 06 '20 at 16:04
  • I’m voting to close this question because This is an old question with no answers, it is unlikely that an answer would help anyone as the issue itself is likely to never occur again – jamesc Jul 22 '20 at 14:55

1 Answers1

1

This was related to a Ruby concurrency issue fixed in a patch for version 2.3

Your logs suggests your app is running 2.0.0. Updating the language version to at least the stable version of 2.3 would fix that problem. This can be easily done by changing your Gemfile Ruby declaration:

ruby '2.3.8'
ErvalhouS
  • 4,178
  • 1
  • 22
  • 38