3

I have a rails app integrated with a third party web service. It has been working perfectly since the initial development for years. For some unexpected reason, suddenly it has stopped working. I would say we haven´t changed anything in our code. The only thing that it can be related is the fact that our Letsencrypt SSL certificate expired and we renewed it.

The fact is that I´m getting this error when calling the web service request:

E, [2022-02-17T19:53:25.385435 #32501] ERROR -- : SSL_connect returned=1 errno=0 state=error: certificate verify failed
E, [2022-02-17T19:53:25.385876 #32501] ERROR -- : /Users/Rober/.rvm/gems/ruby-2.4.9/gems/httpi-2.4.4/lib/httpi/adapter/httpclient.rb:28:in `rescue in request'
E, [2022-02-17T19:53:25.386103 #32501] ERROR -- : /Users/Rober/.rvm/gems/ruby-2.4.9/gems/httpi-2.4.4/lib/httpi/adapter/httpclient.rb:24:in `request'
E, [2022-02-17T19:53:25.386358 #32501] ERROR -- : /Users/Rober/.rvm/gems/ruby-2.4.9/gems/httpi-2.4.4/lib/httpi.rb:161:in `request'
E, [2022-02-17T19:53:25.386658 #32501] ERROR -- : /Users/Rober/.rvm/gems/ruby-2.4.9/gems/httpi-2.4.4/lib/httpi.rb:127:in `get'
E, [2022-02-17T19:53:25.386909 #32501] ERROR -- : /Users/Rober/.rvm/gems/ruby-2.4.9/gems/wasabi-3.5.0/lib/wasabi/resolver.rb:43:in `load_from_remote'
E, [2022-02-17T19:53:25.387150 #32501] ERROR -- : /Users/Rober/.rvm/gems/ruby-2.4.9/gems/wasabi-3.5.0/lib/wasabi/resolver.rb:33:in `resolve'
E, [2022-02-17T19:53:25.387349 #32501] ERROR -- : /Users/Rober/.rvm/gems/ruby-2.4.9/gems/wasabi-3.5.0/lib/wasabi/document.rb:142:in `xml'
E, [2022-02-17T19:53:25.387606 #32501] ERROR -- : /Users/Rober/.rvm/gems/ruby-2.4.9/gems/wasabi-3.5.0/lib/wasabi/document.rb:160:in `parse'
E, [2022-02-17T19:53:25.387887 #32501] ERROR -- : /Users/Rober/.rvm/gems/ruby-2.4.9/gems/wasabi-3.5.0/lib/wasabi/document.rb:147:in `parser'
E, [2022-02-17T19:53:25.388162 #32501] ERROR -- : /Users/Rober/.rvm/gems/ruby-2.4.9/gems/wasabi-3.5.0/lib/wasabi/document.rb:64:in `soap_actions'
E, [2022-02-17T19:53:25.388432 #32501] ERROR -- : /Users/Rober/.rvm/gems/ruby-2.4.9/gems/savon-2.12.0/lib/savon/operation.rb:22:in `ensure_exists!'
E, [2022-02-17T19:53:25.388696 #32501] ERROR -- : /Users/Rober/.rvm/gems/ruby-2.4.9/gems/savon-2.12.0/lib/savon/operation.rb:15:in `create'
E, [2022-02-17T19:53:25.388955 #32501] ERROR -- : /Users/Rober/.rvm/gems/ruby-2.4.9/gems/savon-2.12.0/lib/savon/client.rb:32:in `operation'
E, [2022-02-17T19:53:25.389214 #32501] ERROR -- : /Users/Rober/.rvm/gems/ruby-2.4.9/gems/savon-2.12.0/lib/savon/client.rb:36:in `call'

I´m not skilled at all with certificates. So, I found this post: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed and basically I´m trying almost everything that I see, but nothing is working.

Production enviroment is running a AWS EC2 instance under Ubuntu 14.04.6 LTS (GNU/Linux 3.13.0-36-generic x86_64).

According to what I have read, it might be related to SSL libs in ruby. It make sense, because I have noticed that I´m getting this error when requesting within my webapp using ruby, however if I request using curl like curl --header "Content-Type: text/xml;charset=UTF-8" --data @request.xml https://www.booking-manager.com/cbm_web_service2/services/CBM I get a successful response with data.

Regarding my ruby environment:

rvm info

ruby-2.4.9:

  system:
    uname:        "Linux ip-172-31-20-213 3.13.0-36-generic #63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux"
    name:         "Ubuntu"
    version:      "14.04"
    architecture: "x86_64"
    bash:         "/bin/bash => GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)"
    zsh:          " => not installed"
    remote_path:  "ubuntu/14.04/x86_64"

  rvm:
    version:      "1.29.12 (manual)"
    updated:      "2 months 20 days 19 hours 54 minutes 44 seconds ago"
    path:         "/usr/share/rvm"
    autolibs:     "[4] Allow RVM to use package manager if found, install missing dependencies, install package manager (only OS X)."

  ruby:
    interpreter:  "ruby"
    version:      "2.4.9p362"
    date:         "2019-10-02"
    platform:     "x86_64-linux"
    patchlevel:   "2019-10-02 revision 67824"
    full_version: "ruby 2.4.9p362 (2019-10-02 revision 67824) [x86_64-linux]"

  homes:
    gem:          "/home/ubuntu/.rvm/gems/ruby-2.4.9"
    ruby:         "/usr/share/rvm/rubies/ruby-2.4.9"

  binaries:
    ruby:         "/usr/share/rvm/rubies/ruby-2.4.9/bin/ruby"
    irb:          "/usr/share/rvm/rubies/ruby-2.4.9/bin/irb"
    gem:          "/usr/share/rvm/rubies/ruby-2.4.9/bin/gem"
    rake:         "/home/ubuntu/.rvm/gems/ruby-2.4.9/bin/rake"

  environment:
    PATH:         "/home/ubuntu/.rvm/gems/ruby-2.4.9/bin:/home/ubuntu/.rvm/gems/ruby-2.4.9@global/bin:/usr/share/rvm/rubies/ruby-2.4.9/bin:/usr/share/rvm/bin:/home/ubuntu/.rbenv/plugins/ruby-build/bin:/home/ubuntu/.rbenv/shims:/home/ubuntu/.rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
    GEM_HOME:     "/home/ubuntu/.rvm/gems/ruby-2.4.9"
    GEM_PATH:     "/home/ubuntu/.rvm/gems/ruby-2.4.9:/home/ubuntu/.rvm/gems/ruby-2.4.9@global"
    MY_RUBY_HOME: "/usr/share/rvm/rubies/ruby-2.4.9"
    IRBRC:        "/usr/share/rvm/rubies/ruby-2.4.9/.irbrc"
    RUBYOPT:      ""
    gemset:       ""

My webapp is running under a nginx front-end sending requests to a ruby on rails running under Puma. My puma -version:

puma -v
    /home/ubuntu/.rvm/gems/ruby-2.4.9/gems/puma-4.3.0/lib/puma/puma_http11.so: [BUG] Segmentation fault at 0x00000000000640
    ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-linux]
    
    -- Control frame information -----------------------------------------------
    c:0022 p:-17524110008176 s:0109 e:000108 TOP    [FINISH]
    c:0021 p:---- s:0107 e:000106 CFUNC  :require
    c:0020 p:0115 s:0103 e:000102 METHOD /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55
    c:0019 p:0087 s:0093 e:000092 TOP    /home/ubuntu/.rvm/gems/ruby-2.4.9/gems/puma-4.3.0/lib/puma/server.rb:15 [FINISH]
    c:0018 p:---- s:0091 e:000090 CFUNC  :require
    c:0017 p:0115 s:0087 e:000086 METHOD /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55
    c:0016 p:0007 s:0077 e:000076 TOP    /home/ubuntu/.rvm/gems/ruby-2.4.9/gems/puma-4.3.0/lib/puma/runner.rb:3 [FINISH]
    c:0015 p:---- s:0075 e:000074 CFUNC  :require
    c:0014 p:0115 s:0071 e:000070 METHOD /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55
    c:0013 p:0007 s:0061 e:000060 TOP    /home/ubuntu/.rvm/gems/ruby-2.4.9/gems/puma-4.3.0/lib/puma/cluster.rb:3 [FINISH]
    c:0012 p:---- s:0059 e:000058 CFUNC  :require
    c:0011 p:0115 s:0055 e:000054 METHOD /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55
    c:0010 p:0023 s:0045 e:000044 TOP    /home/ubuntu/.rvm/gems/ruby-2.4.9/gems/puma-4.3.0/lib/puma/launcher.rb:5 [FINISH]
    c:0009 p:---- s:0043 e:000042 CFUNC  :require
    c:0008 p:0115 s:0039 e:000038 METHOD /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55
    c:0007 p:0039 s:0029 e:000028 TOP    /home/ubuntu/.rvm/gems/ruby-2.4.9/gems/puma-4.3.0/lib/puma/cli.rb:8 [FINISH]
    c:0006 p:---- s:0027 e:000026 CFUNC  :require
    c:0005 p:0115 s:0023 e:000022 METHOD /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55
    c:0004 p:0007 s:0013 e:000012 TOP    /home/ubuntu/.rvm/gems/ruby-2.4.9/gems/puma-4.3.0/bin/puma:6 [FINISH]
    c:0003 p:---- s:0010 e:000009 CFUNC  :load
    c:0002 p:0135 s:0006 E:001c18 EVAL   /home/ubuntu/.rbenv/versions/2.1.2/bin/puma:23 [FINISH]
    c:0001 p:0000 s:0002 E:0019f8 TOP    [FINISH]
    
    -- Ruby level backtrace information ----------------------------------------
    /home/ubuntu/.rbenv/versions/2.1.2/bin/puma:23:in `<main>'
    /home/ubuntu/.rbenv/versions/2.1.2/bin/puma:23:in `load'
    /home/ubuntu/.rvm/gems/ruby-2.4.9/gems/puma-4.3.0/bin/puma:6:in `<top (required)>'
    /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    /home/ubuntu/.rvm/gems/ruby-2.4.9/gems/puma-4.3.0/lib/puma/cli.rb:8:in `<top (required)>'
    /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    /home/ubuntu/.rvm/gems/ruby-2.4.9/gems/puma-4.3.0/lib/puma/launcher.rb:5:in `<top (required)>'
    /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    /home/ubuntu/.rvm/gems/ruby-2.4.9/gems/puma-4.3.0/lib/puma/cluster.rb:3:in `<top (required)>'
    /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    /home/ubuntu/.rvm/gems/ruby-2.4.9/gems/puma-4.3.0/lib/puma/runner.rb:3:in `<top (required)>'
    /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    /home/ubuntu/.rvm/gems/ruby-2.4.9/gems/puma-4.3.0/lib/puma/server.rb:15:in `<top (required)>'
    /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
...
    
    [NOTE]
    You may have encountered a bug in the Ruby interpreter or extension libraries.
    Bug reports are welcome.
    For details: http://www.ruby-lang.org/bugreport.html
    
    Aborted (core dumped)

I´m trying to connect to a third party web service:

client = Savon.client(wsdl: "https://www.booking-manager.com/cbm_web_service2/services/CBM", 
                              #log_level: :info,
                              log_level: :debug,
                              log: true,
                              pretty_print_xml: true,
                              open_timeout: 300, 
                              read_timeout: 300)
message = {'in0' => Yanpy::MMK_USER_ID, 
               'in1' => Yanpy::MMK_USERNAME, 
               'in2' => Yanpy::MMK_PASSWORD}
    response = client.call(:get_regions, message: message)

UPDATE According to the answer provided by @jangaraj below, I could fix the problem in development environment: Error Certificate verify failed (certificate has expired)): in Mac OSX 11.6.1 and ruby 3.0.3. However, I still could not fix the problem in production. I think the root cause for this is that before updating my web service request with the right ca-certificates file, I need to have it clear where is this file and if it´s working. For this, according again to the steps that I followed in the other post for development, I run:

openssl s_client -showcerts -host valid-isrgrootx1.letsencrypt.org -port 443

and I could debug and fix the ca-certificates configuration following the steps in https://community.letsencrypt.org/t/help-thread-for-dst-root-ca-x3-expiration-september-2021/149190/970 However, at the moment this is not working for me. I see some messy ca-certs folders and files, probably from different installations? I try to explain:

  • /usr/share/ca-certificates/mozilla : I have a list of .crt files. DST_Root_CA_X3.crt has been removed and ISRG_Root_X1.crt is present.
  • /etc/ca-certificates/update.d is empty.
  • /etc/ca-certificates.conf with a list of ca-certificates. I can see the #mozilla/DST_Root_CA_X3.crt commented and mozilla/ISRG_Root_X1.crt available.
  • The previous file mentions that ca-certs are installed at /etc/ssl/certs where I can see a list of simlinks to the /usr/share/ca-certificates/mozilla folder above. Note that there was a file/link pointing to DST_Root_CA_X3.crt that has been removed. In the other hand, I can see:
  1. ISRG_Root_X1.pem -> /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
  2. 4042bcee.0 -> ISRG_Root_X1.pem
  3. 6187b673.0 -> ISRG_Root_X1.pem
  • but apart from this in /etc/ssl/certs I can also see a ca-certificates.crt with a list of certificates inside in format:

    -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----

  • /usr/lib/ssl/certs seems to have same certificates that /etc/ssl/certs.

The fact is that when I run openssl s_client -showcerts -host valid-isrgrootx1.letsencrypt.org -port 443 I still get next error:

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/CN=origin.letsencrypt.org
   i:/C=US/O=Let's Encrypt/CN=R3
-----BEGIN CERTIFICATE-----
MIIFMTCCBBmgAwIBAgISAz+qxNSV9WDWFPKhpVo1EzpqMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMjAyMjMxNTAwMjFaFw0yMjA1MjQxNTAwMjBaMCExHzAdBgNVBAMT
Fm9yaWdpbi5sZXRzZW5jcnlwdC5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQCZbsEvIzZDb5R7A/pDm4EYksNEMlQEo5uJT+CFA29bb8ldJAcsqdOj
Fge72dr2qWcowC2azODBuS0fE0nMKpt09r+DX/k3HeF2Z9v/HW/B3ManDgpo7FfT
F6m7zyfEowVPK82hm1aruBlBJAcMZrwOvQIfeiOG6hxkH+ScHmoiiVAlWt1v9wKK
tgELlhn5OJkGklVX8xPY+2UlEWaSqoyPmiYnAE+tk4vi6kcAYgpcTOUOrsLnVgAW
iy3ShTAlM8AfGnRYV5dkdzSafRmbi2bJXhoINwi2NDKboZHZY88Au8PNZlA32XBo
E8RBI/bVaK5/rWjN0IJffvv+N5C4kSWLAgMBAAGjggJQMIICTDAOBgNVHQ8BAf8E
BAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQC
MAAwHQYDVR0OBBYEFLIea0+W5J3EbMku+rkQSL92smpuMB8GA1UdIwQYMBaAFBQu
sxe3WFbLrlAJQOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEFBQcwAYYV
aHR0cDovL3IzLm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8vcjMuaS5s
ZW5jci5vcmcvMCEGA1UdEQQaMBiCFm9yaWdpbi5sZXRzZW5jcnlwdC5vcmcwTAYD
VR0gBEUwQzAIBgZngQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcCARYa
aHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwggEDBgorBgEEAdZ5AgQCBIH0BIHx
AO8AdQDfpV6raIJPH2yt7rhfTj5a6s2iEqRqXo47EsAgRFwqcwAAAX8nT+NHAAAE
AwBGMEQCIH1xywyRde3OwoG0RZQW0xfXFEfnd5IFDuSEWHmaxaJEAiAmqTVJrTo1
ycmg1DUrlp087WgXdE8RleCkwWLiiCI0gAB2ACl5vvCeOTkh8FZzn2Old+W+V32c
YAr4+U1dJlwlXceEAAABfydP478AAAQDAEcwRQIgI65GyXpBqx6zYOVRu6jBNVNa
yXR//Rvfih5E6oR8or0CIQD+GA9xLWfIsexfPpqczQgxlKrHHU7Jm9635VjotOJG
wTANBgkqhkiG9w0BAQsFAAOCAQEAgUSK3HvXnKsR+WqNSmJvcXPzEZaTTp1sq4++
zRK6sZ1TJsJQq00aPX1625fSOYNQa0vYDFahfEeXMQ9IceOQJ+HE7NZYgnjnKb3u
96fhRyGBpPEY3szbtJQWk8oDQgFk3u7FS+AnxyXP88ypiC4iDk16Ab9Nip+8sz6t
JOzo3KvVlucviS8K4vJemz6WyH8Ux7KX3r/IvPFj7Cx2aBAx6QM7lXWXgo6PHS/r
eo+KjKo0YWZfdXm7oD7VpOlPNr4W7kAuKidmPSwOPB7RGJ/OKfbGiFCVt8Yy9cyp
hMYLmYJ9qZtxKFGi2kR7Cl/0LuXaBNy4SximhxPUWXQZhoNL3A==
-----END CERTIFICATE-----
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=origin.letsencrypt.org
issuer=/C=US/O=Let's Encrypt/CN=R3
---
No client certificate CA names sent
---
SSL handshake has read 4700 bytes and written 421 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 1DA772568CC98B40026497CB0013ABD22F5F2D9142370B99017A3C84EAEBC0BD
    Session-ID-ctx: 
    Master-Key: C0324E0CDC2FEA59C3A921E1CDCBED19DD7EF2D4785B4BC8208B18934E1E8FA646692F3AAB956CFA2646015AEB3A6AAB
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - a7 64 6c 88 3c 76 89 cf-37 95 cd b6 5f 84 c3 71   .dl.<v..7..._..q
    0010 - de ca 95 74 f3 f8 9f 08-eb bc e7 be a5 63 ca e4   ...t.........c..
    0020 - 1e 80 c9 a6 a7 bd fe 9e-a0 ae f7 f1 64 74 b3 ff   ............dt..
    0030 - e2 8d 1e 2a 51 0a 5a f5-77 6d 86 b6 87 28 a1 2a   ...*Q.Z.wm...(.*
    0040 - e0 ff 79 d8 d5 89 52 99-a7 50 ca 35 62 30 97 f9   ..y...R..P.5b0..
    0050 - 24 57 b3 e5 87 4a 60 04-c2 e9 45 c7 47 cd b9 a9   $W...J`...E.G...
    0060 - b2 d5 f9 82 05 f6 98 5f-54 4a 5e 4a f5 06 66 da   ......._TJ^J..f.
    0070 - e6 ba 13 ff 66 ff a3 3a-b7 1c db fa 05 ad 51 0f   ....f..:......Q.
    0080 - ba ad fe 92 ea e7 c6 92-02 89 ec 83 06 46 06 2d   .............F.-
    0090 - 1b 96 95 81 80 4a eb 55-b1 80 6a 5d e6 09 78 75   .....J.U..j]..xu
    00a0 - fe a9 c2 d8 d2 e2 31 a5-77 c5 d2 e2 c9 3b d0 01   ......1.w....;..

    Start Time: 1646907242
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---

so, DST Root CA X3 seems to be present somewhere. I´m stuck at this point.

openssl version
OpenSSL 1.0.1f 6 Jan 2014

If I run:

ubuntu@ip-172-31-9-63:~$ irb
2.1.4 :001 > require "openssl"
 => true 
2.1.4 :002 > puts OpenSSL::OPENSSL_VERSION
OpenSSL 1.0.1f 6 Jan 2014
 => nil 
2.1.4 :003 > puts "SSL_CERT_FILE: %s" % OpenSSL::X509::DEFAULT_CERT_FILE
SSL_CERT_FILE: /usr/lib/ssl/cert.pem
 => nil 
2.1.4 :004 > puts "SSL_CERT_DIR: %s" % OpenSSL::X509::DEFAULT_CERT_DIR
SSL_CERT_DIR: /usr/lib/ssl/certs
 => nil 

However /usr/lib/ssl/cert.pem does not exist.

Also, if I do:

openssl version -d

I get: OPENSSLDIR: "/usr/lib/ssl"

UPDATE To separate the issues, I have created a new post with the current issue here: Removing old Digital Signature Trust Co./CN=DST Root CA X3 throwing error verify error:num=20:unable to get local issuer certificate in Ubuntu Part 1 I think once it´s fixed, I will be able to fix the one in this post.

Rober
  • 5,868
  • 17
  • 58
  • 110
  • 2
    Details are very scarce, i.e. nearly nothing known about your server and client and because of that no way to dig deeper and reproduce the problem. But since the error occurred after you've changed the certificate then the certificate installation might have been done wrong. Typical issues are missing intermediate certificates. Check your site with [SSLLabs](https://www.ssllabs.com/ssltest/analyze.html) and look out for problems, specifically "chain issues". – Steffen Ullrich Feb 17 '22 at 19:54
  • Hi, I have add some updated information. Please let me know if you need some additional details and I will provide them. Please note that I cannot check the site with SSLLabs because I´m trying to fix my localhost development enviroment. – Rober Feb 20 '22 at 19:51
  • *"... because I´m trying to fix my localhost development enviroment."* - So you are using a self-signed certificate I suppose. A self-signed certificate is not issued by a publicly trusted CA though, which means certificate validation will fail if you only trust the standard sets of CA. – Steffen Ullrich Feb 20 '22 at 19:57
  • In production, we used a Letsencrypt certificate. To be honest, I´m not sure if I did the same in development. Also, I don´t know if I installed self-signed certificate, can you please let me know how can I check it and I will let you know? What is a fact, is that something has changed also in development, because I didn´t get this error before. – Rober Feb 20 '22 at 20:04
  • *"In production, we used a Letsencrypt certificate"* - if the problem exists in production then you should be able to check the server with SSLLabs (assuming a publicly reachable server). *"can you please let me know how can I check it"* - given that you are not skilled at all in this topic (as you say yourself) this will likely be too complex. Better focus on the problems with the production setup first which can be debugged with SSLLabs. – Steffen Ullrich Feb 20 '22 at 20:13
  • This is the domain name in prod: www.yanpy.com. I already passed the test and I´m not able to analize/identify the problem. I can only see: DNS CAA No (more info). As said, I´m not skilled at all, but my concern is that I wonder if I´m mixing two certificates. Letsencrypt for HTTPS and other certificates used by Ruby. I think this is related to ruby certificates. – Rober Feb 20 '22 at 20:24
  • The site is fine. Not much is known about your environment but it might be affected by the recent change in lets encrypt certificates, notably that one intermediate CA expired. Maybe the answer https://stackoverflow.com/a/69408777/3081018 can help you with this. – Steffen Ullrich Feb 20 '22 at 20:30
  • Hi, I have updated the information to provide more details. – Rober Mar 01 '22 at 04:37
  • It looks like a problem with old CA certificates - but you didn't list them, so it is just guess. You can try to update CA certificates of your Ubuntu: `sudo apt-get update -y; apt --only-upgrade install -y ca-certificates` – Jan Garaj Mar 07 '22 at 12:14
  • I totally agree. Although probably it´s easier to debug the production environment, I don´t think it´s very good idea to play without much knowledge directly in it. So, I just created this similar post with same issue but in my Mac OSX development environment. Please read this: https://stackoverflow.com/questions/71373958/error-certificate-verify-failed-certificate-has-expired-in-mac-osx-11-6-1-an – Rober Mar 07 '22 at 19:27

3 Answers3

5

It looks like you are using Ubuntu 14 and Savon 2 client. Savon 2 client doc: https://www.savonrb.com/version2/globals.html

ssl_ca_cert_file

Sets the SSL ca cert file to use.

Savon.client(ssl_ca_cert_file: "lib/ca_cert.pem")

I would point ssl_ca_cert_file to /etc/ssl/certs/ca-certificates.crt explicitly.

To make sure that your OS has valid CA certs:

apt-get update -y
apt --only-upgrade install -y ca-certificates
update-ca-certificates

There can be also a problem with old OpenSSL version used by Ruby.

Jan Garaj
  • 25,598
  • 3
  • 38
  • 59
  • OMG this is working in development. Let me check in production and get back. – Rober Mar 08 '22 at 04:34
  • @Rober is it working? – Jan Garaj Mar 09 '22 at 06:52
  • As I mentioned, I could fix it in development. Actually, your answer, should fix my other development post: https://stackoverflow.com/questions/71373958/error-certificate-verify-failed-certificate-has-expired-in-mac-osx-11-6-1-an I will update it with your answer. Or just answer there if you want. Regarding the issue in production, unfortunately I´m facing some problems that could fix in development and not working in prod. I will detail them above. – Rober Mar 10 '22 at 09:39
  • Hello, I have updated the information above. – Rober Mar 10 '22 at 10:35
  • I added some additional information. From all different ca-certs folders, how can I check where is openssl getting the ca-certs? – Rober Mar 10 '22 at 15:33
  • I could not fix the problem yet. In order to try to narrow the issue down, let´s go step by step. The problem at this moment is that when I run openssl s_client -showcerts -host valid-isrgrootx1.letsencrypt.org -port 443 I´m still getting Verify return code: 20 (unable to get local issuer certificate) because it is still finding the Digital Signature Trust Co./CN=DST Root CA X3. But I don´t know where is it getting it from. I updated my list of ca certs folders above. Until I don´t fix this, I won´t be able to apply @jangaraj solution. – Rober Mar 25 '22 at 05:33
  • I have updated this post and created a new one (I added the link above) with the current problem. – Rober Mar 25 '22 at 06:50
1

-Solutions Update CA certificates The correct solution depends on which code connects to an HTTPS URL. The first thing you can try is to update the root certificates on your machine.

If you’re using Linux, you can use your package manager to update the CA certificates.

apt-get update ca-certificates
yum update ca-certificates

On RVM on OSX, you can run

rvm osx-ssl-certs update all

If you don’t use RVM, you can extract the certificates from Apple’s Keychain yourself.

cert_file='$( openssl version -d | awk -F''' '{print $2}' )/cert.pem'
mkdir -p '${cert_file%/*}'
security find-certificate -a -p /Library/Keychains/System.keychain > '$cert_file'
security find-certificate -a -p 
/System/Library/Keychains/SystemRootCertificates.keychain >> '$cert_file'

You can check out the SSL documentation.

Thanks

Tyler Minegar
  • 317
  • 1
  • 5
  • 1
    As mentioned above, I´m using Ubuntu. When I run apt-get update ca-certificates I get E: The update command takes no arguments – Rober Mar 25 '22 at 05:36
  • I have updated this post and created a new one (I added the link above) with the current problem. – Rober Mar 25 '22 at 06:50
0

One thing you might check is that your certs directory (often /etc/ssl/certs ) contains both your certificate AND a symbolic link to that cert (based on a hash of the certificate).

openssl uses the symbolic link to look up the certificate and other certificates up the chain. If you do a "ls -l " in your certs directory, you ought to see many lines something like this:

lrwxrwxrwx 1 root root     18 Dec  2  2017 f39fc864.0 -> SecureTrust_CA.pem

The file name f39fc864 is a hash of the certificate, and the .0 suffix is required. If your certificate does not have such a link, and even if it does, you can enter:

openssl x509 -noout -hash -in mycertfile.pem

and it will generate an 8-digit hash, like 23cd45d6.

After you generate the hash, you would just create the link using a command that looks like this (where you would substitute your file and your hash, obviously):

ln -s mycertfile.pem  23cd45d6.0

In fact openssl has a command "c_rehash" that will do this for you. The man page for openssl c_rehash will guide you. But on many systems it will be as simple as

openssl c_rehash /etc/ssl/certs

if that is where your certs are located.

  • As you can read in the main description above, I have a mess of ca-certs files and directory. Would be one step to know which one are the valid ones. Is it possible to know where is openssl getting the certs? I mean which directory/files. – Rober Mar 25 '22 at 05:39
  • I have updated this post and created a new one (I added the link above) with the current problem. – Rober Mar 25 '22 at 06:50