103

I am aware that Let's Encrypt made changes that may impact older clients because a root certificate would expire. See DST Root CA X3 Expiration (September 2021).

However, I didn't think this could impact me because my development machine is up-to-date.

But since today I get the message while doing a git pull:

fatal: unable to access 'https://git.company.tld/project.git/': SSL certificate problem: certificate has expired

I just downloaded the newest Git for Windows (2.33.0) and confirmed that the built-in OpenSSL is up-to-date (OpenSSL 1.1.1k 25 Mar 2021) which should be good.

OpenSSL Client Compatibility Changes for Let’s Encrypt Certificates

But the error seems to stay.

openssl s_client -showcerts -connect git.company.tld:443

shows

CONNECTED(000001A0)
---
Certificate chain
 0 s:CN = git.company.tld
   i:C = US, O = Let's Encrypt, CN = R3
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
---
Server certificate
subject=CN = git.company.tld

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3058 bytes and written 443 bytes
Verification error: certificate has expired
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: ...
    Session-ID-ctx:
    Master-Key: ...
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1632982992
    Timeout   : 7200 (sec)
    Verify return code: 10 (certificate has expired)
    Extended master secret: no
---

The problem is not with the issued certificate itself which is not expired and accepted by Chrome (Windows certificate store) and Firefox.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Jürgen Steinblock
  • 30,746
  • 24
  • 119
  • 189
  • Similar topic on SuperUser: [cURL on Ubuntu 14: all Let's Encrypt certificates are expired (error 60)](https://superuser.com/questions/1679204/curl-on-ubuntu-14-all-lets-encrypt-certificates-are-expired-error-60) – ElementW Oct 02 '21 at 11:00
  • This is my question still unanswered. I am also using Let's Encrypt . Can it be related to this?https://stackoverflow.com/questions/69400588/phonegap-ajax-request-error-certificate-has-expired – dspillai Oct 02 '21 at 17:45
  • Git for Windows 2.33.0 should work. You might want to check with "git config -l | grep ssl" if you have set sslCAInfo to something other then "C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.cr" – nharrer Oct 02 '21 at 19:56
  • 2
    After spending hours try fixing this I gave up: `git config http.sslVerify false` . – michid Oct 04 '21 at 09:50

12 Answers12

63

On my Ubuntu 16.04.6 LTS (Xenial Xerus) machine, the solution was to remove the DST Root CA X3 certificate by running the following two commands:

sudo rm /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt
sudo update-ca-certificates

On my Mac OS X 10.13.6 (High Sierra) machine, cURL (and therefore Git) rely on the /etc/ssl/cert.pem file for root CA verification.

The solution was to remove the DST Root CA X3 certificate, which expired today, from the file:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
        Validity
            Not Before: Sep 30 21:12:19 2000 GMT
            Not After : Sep 30 14:01:15 2021 GMT
        Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:df:af:e9:97:50:08:83:57:b4:cc:62:65:f6:90:
                    82:ec:c7:d3:2c:6b:30:ca:5b:ec:d9:c3:7d:c7:40:
                    c1:18:14:8b:e0:e8:33:76:49:2a:e3:3f:21:49:93:
                    ac:4e:0e:af:3e:48:cb:65:ee:fc:d3:21:0f:65:d2:
                    2a:d9:32:8f:8c:e5:f7:77:b0:12:7b:b5:95:c0:89:
                    a3:a9:ba:ed:73:2e:7a:0c:06:32:83:a2:7e:8a:14:
                    30:cd:11:a0:e1:2a:38:b9:79:0a:31:fd:50:bd:80:
                    65:df:b7:51:63:83:c8:e2:88:61:ea:4b:61:81:ec:
                    52:6b:b9:a2:e2:4b:1a:28:9f:48:a3:9e:0c:da:09:
                    8e:3e:17:2e:1e:dd:20:df:5b:c6:2a:8a:ab:2e:bd:
                    70:ad:c5:0b:1a:25:90:74:72:c5:7b:6a:ab:34:d6:
                    30:89:ff:e5:68:13:7b:54:0b:c8:d6:ae:ec:5a:9c:
                    92:1e:3d:64:b3:8c:c6:df:bf:c9:41:70:ec:16:72:
                    d5:26:ec:38:55:39:43:d0:fc:fd:18:5c:40:f1:97:
                    eb:d5:9a:9b:8d:1d:ba:da:25:b9:c6:d8:df:c1:15:
                    02:3a:ab:da:6e:f1:3e:2e:f5:5c:08:9c:3c:d6:83:
                    69:e4:10:9b:19:2a:b6:29:57:e3:e5:3d:9b:9f:f0:
                    02:5d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Subject Key Identifier:
                C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10
    Signature Algorithm: sha1WithRSAEncryption
         a3:1a:2c:9b:17:00:5c:a9:1e:ee:28:66:37:3a:bf:83:c7:3f:
         4b:c3:09:a0:95:20:5d:e3:d9:59:44:d2:3e:0d:3e:bd:8a:4b:
         a0:74:1f:ce:10:82:9c:74:1a:1d:7e:98:1a:dd:cb:13:4b:b3:
         20:44:e4:91:e9:cc:fc:7d:a5:db:6a:e5:fe:e6:fd:e0:4e:dd:
         b7:00:3a:b5:70:49:af:f2:e5:eb:02:f1:d1:02:8b:19:cb:94:
         3a:5e:48:c4:18:1e:58:19:5f:1e:02:5a:f0:0c:f1:b1:ad:a9:
         dc:59:86:8b:6e:e9:91:f5:86:ca:fa:b9:66:33:aa:59:5b:ce:
         e2:a7:16:73:47:cb:2b:cc:99:b0:37:48:cf:e3:56:4b:f5:cf:
         0f:0c:72:32:87:c6:f0:44:bb:53:72:6d:43:f5:26:48:9a:52:
         67:b7:58:ab:fe:67:76:71:78:db:0d:a2:56:14:13:39:24:31:
         85:a2:a8:02:5a:30:47:e1:dd:50:07:bc:02:09:90:00:eb:64:
         63:60:9b:16:bc:88:c9:12:e6:d2:7d:91:8b:f9:3d:32:8d:65:
         b4:e9:7c:b1:57:76:ea:c5:b6:28:39:bf:15:65:1c:c8:f6:77:
         96:6a:0a:8d:77:0b:d8:91:0b:04:8e:07:db:29:b6:0a:ee:9d:
         82:35:35:10
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----

After removing the entire code snippet above from the file and saving it, the error went away.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Elad Nava
  • 7,746
  • 2
  • 41
  • 61
19

I had the same issue because I was running an old version of Git for Windows (2.15.0). After updating Git to the latest version (2.33) the problem was fixed.

Reason: Older versions of Git would not accept the expired root certificate from Let's Encrypt.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
15

I faced the same problem on an Ubuntu 14.04 LTS (Trusty Tahr) server.

This problem affected cURL calls from PHP, etc.

Example: from a simple curl on the command line,

curl https://curl.se/ca/cacert.pem

I was getting the certificate expired message. It was due to the old Let's Encrypt certificate expiration. dst-root-ca-x3-expiration-september-2021

So, here is a simple working solution:

Edit the file /etc/ca-certificates.conf.

Find and comment the lines with ! like this:

!mozilla/DST_Root_CA_X3.crt

Or run the following command to automatically comment the certificate:

sudo sed -i -e 's/mozilla\/DST_Root_CA_X3\.crt/!mozilla\/DST_Root_CA_X3\.crt/g' /etc/ca-certificates.conf

Save the file

Update certificates:

sudo update-ca-certificates

And all is done... It's working, and there isn't any need to change other things...

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
  • Hi, thanks, does this solution have any disadvantages for a website to which visitors connect? – user2267379 Oct 01 '21 at 17:55
  • @christian audebert thank you very much, you saved my headache. I was so close I firstly tried manually remove /etc/ssl/certs/DST_Root_CA_X3.crt but did not help and I reverted it back and secondly deleted it from /etc/ssl/ca-certificates.crt but it did not help too (my bad, I had to do both steps to solve issue for testing). Of course correct way is to edit /etc/ca-certificates.conf but I wanted conservatively do quick test if it helps without doing update-ca-certificates. Now curl via terminal and curl via PHP scripts can access lets encrypt-ed websites :) – mikep Oct 01 '21 at 18:23
  • @user2267379 this does not affect website visitors, this change is only for server which acts as client when connecting to HTTPS website (e.g. using curl, wget, etc.) – mikep Oct 01 '21 at 18:26
  • @mekip : some visitors can no longer connect to the site since the change, which solution works for them? – user2267379 Oct 02 '21 at 17:10
  • Tried the same with ubuntu 16.04.2 to no avail. Would you have another idea? I've tried so many things but curl won't run correectly anymore.. – Shautieh Oct 04 '21 at 17:19
  • comment line as in `#!mozilla/DST_Root_CA_X3.crt` right? because in my file it was already `!mozilla/DST_Root_CA_X3.crt` – Shaho Oct 19 '21 at 14:04
  • Adding a comment in case someone ends up here looking for my same problem; I was getting "CCurlFile::FillBuffer - Failed: Peer certificate cannot be authenticated with given CA certificates(60)" errors on Kodi (OSMC) when fetching artwork. This solved the problem! – Zep Oct 23 '21 at 20:56
13

On our Windows test clients we had to update Git to the latest version.

Since Git 2.16.1(2) you can use

git update-git-for-windows

In version between 2.14.2 and 2.16.1, the command was

git update

See also: How to upgrade Git on Windows to the latest version

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
zerocewl
  • 11,401
  • 6
  • 27
  • 53
6

For those who have servers running on Ubuntu, with Certbot managing certificates, I have forced the renewals using ISRG Root X1. This way, new certificates don't contain the chain of DST Root CA X3, and this did the trick for us.

To do that, first check if your Certbot version is < 1:

sudo certbot --version

If so, you have to remove it and reinstall using Snap:

sudo apt-get remove -y certbot python3-certbot-apache
sudo snap install certbot --classic
sudo ln -s /snap/bin/certbot /usr/bin/certbot

After reinstalling, or if your Certbot version is > 1, force the renewal:

sudo certbot renew --force-renewal --preferred-chain "ISRG Root X1"

I also have used DigiCert SSL Installation Diagnostics Tool to check my certificates, before and after renewing, to verify if the DST X3 chain was removed.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Felix Runye
  • 2,135
  • 1
  • 20
  • 20
  • 1
    Thanks a lot! Worked perfectly for my ubuntu 14 apache server. curl error (error 60) is now gone. – paperboy Oct 03 '21 at 02:41
4

Apparently this is not a client issue, but the Let's Encrypt certificate being served by a Sophos UTM WAF (latest version, 9.707-5).

The same certificate served from an Apache web server works fine (and the openssl s_client -showcerts response looks different -> more entries in the certificate chain). So this is not a client-related problem.

Update

The solution was to delete the old Lets Encrypt CA (48:50:4E:97:...)

CA

from Webserver ProtectionCertificate ManagementCertificate Authority.

Client requests worked instantly.

Thanks to VolkerZier in the Sophos forum for giving the hint. See (in German) Let's Encrypt Root Zertifikat gültig bis 30.09.2021 (alte R3 / X3 Zertifikatskette).

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Jürgen Steinblock
  • 30,746
  • 24
  • 119
  • 189
4

For applications based on OpenSSL <= 1.0.2 such as Ubuntu 12.04 (Precise Pangolin), you need to allow OpenSSL to use the alternate chain path to trust the remote site.

First you need to install the ISRG_Root_X1.crt certificate and remove the expired one from the trusted store: DST_Root_CA_X3.crt

This will allow that clients using OpenSSL like Wget, cURL, etc. to work again. The steps are:

To install the ISRG_Root_X1 certificate:

sudo wget http://launchpadlibrarian.net/482351465/ca-certificates_20190110~12.04.1_all.deb
sudo dpkg -i ca-certificates_20190110~12.04.1_all.deb

To disable the DST_Root_CA_X3 certificate:

sudo vi /etc/ca-certificates.conf

Then set: !mozilla/DST_Root_CA_X3.crt

Note: In this file, when the line begins with # is comment. When the line begins with ! is certificate filename to be deselected.

And finally run:

sudo update-ca-certificates

After that, wget and curl will not complain any more.

The best explanation I've found out there is the video DST Root CAX3 Expiration Sept 2021 (34 minutes).

frommelmak
  • 169
  • 3
  • What is "`!`"? A comment character? To outcomment (disable)? Can you clarify? Please respond by [editing (changing) your answer](https://stackoverflow.com/posts/69405794/edit), not here in comments (***without*** "Edit:", "Update:", or similar - the answer should appear as if it was written today). – Peter Mortensen Oct 17 '21 at 14:09
  • so change it to `#!mozilla/DST_Root_CA_X3.crt` ?? – Shaho Oct 19 '21 at 13:03
  • No. Just "!" without "#" cause the cert will be deselected. – frommelmak Oct 22 '21 at 06:08
2

I was facing a similar issue with DevOps build agents. But I can access the DevOps server web interface without any issue.

To solve this,

  • I updated my Let's Encrypt client (I'm using Certify The Web)
  • I have renewed my certificate

After that, the DevOps agent is able to do a Git pull.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
2

Verify the version of GnuTLS (libgnutls). The issue with alternate chains was fixed in 3.6.13-4. So updating GnuTLS to a version above this might solve the issue for Git.

Backstory:

(On Debian systems at least) curl/wget uses libssl/OpenSSL and Git uses libgnutls30 via libcurl3-gnutls.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Vipin Menon
  • 2,892
  • 4
  • 20
  • 35
0

I (as will many in the Third World), have several Windows XP 32/64-bit machines. They are since Friday giving me the "YOUR CLOCK IS AHEAD" error, which is part, I believe, of this SSL certificate lapse. (They were all flawless, universally across the web with Chrome previously).

Many websites (~40%) I visit on the Windows XP machines (handy for legacy software, etc), all give the same TIME error-msg.

My instinct tells me:

Those new certificates have to be acquired, installed and become active and that's going to TAKE TIME to propagate system-wide, meaning ... globally.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
0

The same issue here. For me, it helps to update Visual Studio 2017.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Oct 05 '21 at 12:52
  • 1
    *"update Visual Studio 2017"* or *"update* ***to*** *Visual Studio 2017"* (you have already left out a lot of words)? If the latter, from which version (e.g., [Visual Studio 2005](https://en.wikipedia.org/wiki/Microsoft_Visual_Studio#2005))? Even if it is patching the installed version, can you make it clearer (e.g. how was it updated and to which exact version number)? Please respond by [editing (changing) your answer](https://stackoverflow.com/posts/69449782/edit), not here in comments (***without*** "Edit:", "Update:", or similar - the answer should appear as if it was written today). – Peter Mortensen Oct 17 '21 at 14:29
0

curl <url> -X POST -H 'authorization: Bearer xxx' [other option]

curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

curl <url> -X POST -H 'authorization: Bearer xxx' [other option] -k

-k, --insecure

(TLS SFTP SCP) By default, every secure connection curl makes is verified to be secure before the transfer takes place. This option makes curl skip the verification step and proceed without checking. curl/manpage

山茶树和葡萄树
  • 2,050
  • 1
  • 18
  • 18