5

We have a process (Rake task) that runs for a long period of time, making HTTPS requests to remote hosts using Faraday with the net_http_persistent adapter. After several hours to days of runtime, it stops making requests.

The process is also making some requests using Excon (to report exceptions to our exception-reporting service), and both HTTP clients are logging the same error, with slight difference in wording. The two errors are:

Faraday::SSLError: SSL_connect SYSCALL returned=5 errno=0 state=error: certificate verify failed

Excon::Error::Certificate: SSL_connect SYSCALL returned=5 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:

Our exception-reporting service is failing to receive any of these exceptions. So we only found these errors in the log.

Community
  • 1
  • 1
Robin Daugherty
  • 7,115
  • 4
  • 45
  • 59

1 Answers1

4

In order to reproduce this issue, I ran a loop of Faraday requests until it failed. What I found is that while the exception raised was Faraday::SSLError, once I tried to perform any other action in the REPL that required the opening of a file, it raised Errno::EMFILE: Too many open files.

Finally I had my clue. The red herring was caused by the OpenSSL library catching the EMFILE system error, and instead raising a general SSL connection failure. Both Faraday and Excon (used by the exception-reporting tool) did this, making it impossible to see the real problem.

The underlying problem was that the process had reached the limit of open files. This was caused by the way that net-http-persistent keeps connections open until instructed to shut down combined with the fact that Faraday has no mechanism to instruct its adapter to shut down.

The solution was switching from net-http-persistent to Excon, which supports persistent connections as long as it's configured to do so:

Faraday.new(url: url) do |faraday|
  faraday.adapter :excon, persistent: true
end
Robin Daugherty
  • 7,115
  • 4
  • 45
  • 59
  • 1
    Glad to hear that worked for you, and thanks for taking the time to share what you found. If you have further excon questions, just let me know and I'll do my best to help. – geemus Jul 26 '19 at 13:33