Apache 2 Mutual TLS Authentication Only Working with "SSLClientVerify optional_no_ca"

Question

I am trying to use Apache2 to provide a REST-API, with mutual TLS Authentication.

If l set the SSLVerifyClient option to require, l don't get the client certificate due to the SSL connection not being established due to what looks like the Server/ Client certificate validation. I am using internal company Test Certificates, and the associated CA Certificate chain.

If l set the SSLVerifyClient option to optional_no_ca, l am able to get connectivity from both Chrome/ CURL (test tools), and get the Certificate information. This allows for only openssl_x509_parse($_SERVER['SSL_CLIENT_CERT']) to check the CN value as authentication.

Questions are:

Is there an issue with the way that l have set up the Apache Certificates that is causing the Certification validation errors? Or an issue with the certificates?
Is there a way for me to test the Server/ Client certificate integration outside of Apache2/ curl?
Is getting the Client Certificate, and check for the CN value sufficient to ensure that nobody is masquerading as the Client?

Note: in the Apache Log files l can see Port 443, although l have set up SSL on Port 8447 (due to Port 443 being blocked). Has this got anything to do with it?

Testing with Chrome

This site can’t provide a secure connection
machine.xyz.com didn’t accept your login certificate, or one may not have been provided.
Try contacting the system admin.
ERR_BAD_SSL_CLIENT_AUTH_CERT

Apache Log

[Thu Mar 04 15:44:24.975602 2021] [ssl:info] [pid 24140:tid 140137567536896] [client 10.65.65.199:59523] AH01964: Connection to child 320 established (server machine.xyz.com:443)
[Thu Mar 04 15:44:24.976219 2021] [ssl:debug] [pid 24140:tid 140137567536896] ssl_engine_kernel.c(2353): [client 10.65.65.199:59523] AH02043: SSL virtual host for servername machine.xyz.com found
[Thu Mar 04 15:44:24.984844 2021] [ssl:debug] [pid 24140:tid 140137567536896] ssl_engine_kernel.c(2236): [client 10.65.65.199:59523] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
[Thu Mar 04 15:44:24.985009 2021] [ssl:debug] [pid 24140:tid 140137567536896] ssl_engine_kernel.c(383): [client 10.65.65.199:59523] AH02034: Initial (No.1) HTTPS request received for child 320 (server machine.xyz.com:443)
[Thu Mar 04 15:44:24.985223 2021] [ssl:debug] [pid 24140:tid 140137567536896] ssl_engine_kernel.c(746): [client 10.65.65.199:59523] AH02255: Changed client verification type will force renegotiation
[Thu Mar 04 15:44:24.985257 2021] [ssl:info] [pid 24140:tid 140137567536896] [client 10.65.65.199:59523] AH02221: Requesting connection re-negotiation
[Thu Mar 04 15:44:24.985287 2021] [ssl:debug] [pid 24140:tid 140137567536896] ssl_engine_kernel.c(975): [client 10.65.65.199:59523] AH02260: Performing full renegotiation: complete handshake protocol (client does support secure renegotiation)
[Thu Mar 04 15:44:24.985367 2021] [ssl:info] [pid 24140:tid 140137567536896] [client 10.65.65.199:59523] AH02226: Awaiting re-negotiation handshake
[Thu Mar 04 15:44:24.991206 2021] [ssl:error] [pid 24140:tid 140137567536896] [client 10.65.65.199:59523] AH02261: Re-negotiation handshake failed
[Thu Mar 04 15:44:24.991293 2021] [ssl:debug] [pid 24140:tid 140137567536896] ssl_engine_io.c(1370): (70014)End of file found: [client 10.65.65.199:59523] AH02007: SSL handshake interrupted by system [Hint: Stop button pressed in browser?!]
[Thu Mar 04 15:44:24.991321 2021] [ssl:info] [pid 24140:tid 140137567536896] [client 10.65.65.199:59523] AH01998: Connection closed to child 320 with abortive shutdown (server machine.xyz.com:443)
[Thu Mar 04 15:44:26.471743 2021] [ssl:info] [pid 24140:tid 140137559144192] [client 10.65.65.199:59526] AH01964: Connection to child 321 established (server machine.xyz.com:443)
[Thu Mar 04 15:44:26.471984 2021] [ssl:debug] [pid 24140:tid 140137559144192] ssl_engine_kernel.c(2353): [client 10.65.65.199:59526] AH02043: SSL virtual host for servername machine.xyz.com found
[Thu Mar 04 15:44:26.477354 2021] [ssl:debug] [pid 24140:tid 140137559144192] ssl_engine_kernel.c(2236): [client 10.65.65.199:59526] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
[Thu Mar 04 15:44:26.478078 2021] [ssl:debug] [pid 24140:tid 140137559144192] ssl_engine_kernel.c(383): [client 10.65.65.199:59526] AH02034: Initial (No.1) HTTPS request received for child 321 (server machine.xyz.com:443)
[Thu Mar 04 15:44:26.478210 2021] [ssl:debug] [pid 24140:tid 140137559144192] ssl_engine_kernel.c(746): [client 10.65.65.199:59526] AH02255: Changed client verification type will force renegotiation
[Thu Mar 04 15:44:26.478245 2021] [ssl:info] [pid 24140:tid 140137559144192] [client 10.65.65.199:59526] AH02221: Requesting connection re-negotiation
[Thu Mar 04 15:44:26.478272 2021] [ssl:debug] [pid 24140:tid 140137559144192] ssl_engine_kernel.c(975): [client 10.65.65.199:59526] AH02260: Performing full renegotiation: complete handshake protocol (client does support secure renegotiation)
[Thu Mar 04 15:44:26.478334 2021] [ssl:info] [pid 24140:tid 140137559144192] [client 10.65.65.199:59526] AH02226: Awaiting re-negotiation handshake
[Thu Mar 04 15:44:26.490828 2021] [ssl:debug] [pid 24140:tid 140137559144192] ssl_engine_kernel.c(1741): [client 10.65.65.199:59526] AH02275: Certificate Verification, depth 1, CRL checking mode: none (0) [subject: CN=XYZ Co Server TEST CA 13,OU=PKI,O=XYZ Co AG,C=DE / issuer: CN=XYZ Co Group Root TEST CA 13,OU=PKI,O=XYZ Co AG,C=DE / serial: 04 / notbefore: Mar 18 00:00:00 2018 GMT / notafter: Oct 18 00:00:00 2026 GMT]
[Thu Mar 04 15:44:26.490898 2021] [ssl:info] [pid 24140:tid 140137559144192] [client 10.65.65.199:59526] AH02276: Certificate Verification: Error (20): unable to get local issuer certificate [subject: CN=XYZ Co Server TEST CA 13,OU=PKI,O=XYZ Co AG,C=DE / issuer: CN=XYZ Co Group Root TEST CA 13,OU=PKI,O=XYZ Co AG,C=DE / serial: 04 / notbefore: Mar 18 00:00:00 2018 GMT / notafter: Oct 18 00:00:00 2026 GMT]
[Thu Mar 04 15:44:26.491006 2021] [ssl:error] [pid 24140:tid 140137559144192] [client 10.65.65.199:59526] AH02261: Re-negotiation handshake failed
[Thu Mar 04 15:44:26.491074 2021] [ssl:error] [pid 24140:tid 140137559144192] SSL Library Error: error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed
[Thu Mar 04 15:44:26.491140 2021] [ssl:info] [pid 24140:tid 140137559144192] [client 10.65.65.199:59526] AH02008: SSL library error 1 in handshake (server machine.xyz.com:443)
[Thu Mar 04 15:44:26.491164 2021] [ssl:info] [pid 24140:tid 140137559144192] SSL Library Error: error:140800FF:SSL routines:ssl3_accept:unknown state
[Thu Mar 04 15:44:26.491181 2021] [ssl:info] [pid 24140:tid 140137559144192] [client 10.65.65.199:59526] AH01998: Connection closed to child 321 with abortive shutdown (server machine.xyz.com:443)

Testing Using Curl

$ curl -v --cacert 'XYZ Company TEST.crt' --cert client-machine.xyz.com_Testing-DoD-Application.crt --key client-machine.xyz.com_Testing-DoD-Application.key https://machine.xyz.com:8447/data-requests/secure/test.php
. .  .. 


. . . .
s://machine.xyz.com:8447/data-requests/secure/test.php
*   Trying 10.99.99.99...
* TCP_NODELAY set
* Connected to machine.xyz.com (10.99.99.99) port 8447 (#0)
* schannel: SSL/TLS connection with machine.xyz.com port 8447 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 183 bytes...
* schannel: sent initial handshake data: sent 183 bytes
* schannel: SSL/TLS connection with machine.xyz.com port 8447 (step 2/3)
* schannel: encrypted data got 4096
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: encrypted data length: 3998
* schannel: encrypted data buffer: offset 3998 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with machine.xyz.com port 8447 (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 5022 length 5022
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with machine.xyz.com port 8447 (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 6046 length 6046
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with machine.xyz.com port 8447 (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 7070 length 7070
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with machine.xyz.com port 8447 (step 2/3)
* schannel: encrypted data got 639
* schannel: encrypted data buffer: offset 7709 length 8094
* schannel: sending next handshake data: sending 126 bytes...
* schannel: SSL/TLS connection with machine.xyz.com port 8447 (step 2/3)
* schannel: encrypted data got 51
* schannel: encrypted data buffer: offset 51 length 8094
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with machine.xyz.com port 8447 (step 3/3)
* schannel: stored credential handle in session cache
> GET /data-requests/secure/test.php HTTP/1.1
> Host: machine.xyz.com:8447
> User-Agent: curl/7.55.1
> Accept: */*
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 33
* schannel: encrypted data buffer: offset 33 length 103424
* schannel: decrypted data length: 0
* schannel: decrypted data added: 0
* schannel: decrypted data cached: offset 0 length 102400
* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection with gmcctu01.uk.db.com port 8447 (step 2/3)
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: sending next handshake data: sending 219 bytes...
* schannel: SSL/TLS connection with gmcctu01.uk.db.com port 8447 (step 2/3)
* schannel: encrypted data got 7969
* schannel: encrypted data buffer: offset 7969 length 103424
* schannel: sending next handshake data: sending 3601 bytes...
* schannel: SSL/TLS connection with machine.xyz.com port 8447 (step 2/3)
* schannel: encrypted data got 31
* schannel: encrypted data buffer: offset 31 length 103424
* schannel: next InitializeSecurityContext failed: SEC_E_UNTRUSTED_ROOT (0x80090325) - The certificate chain was issued by an authority that is not trusted.
* schannel: renegotiation failed
* schannel: schannel_recv cleanup
* Closing connection 0
* schannel: shutting down SSL/TLS connection with machine.xyz.com port 8447
* schannel: clear security context handle
curl: (77) schannel: next InitializeSecurityContext failed: SEC_E_UNTRUSTED_ROOT (0x80090325) - The certificate chain was issued by an authority that is not trusted.

Apache Log

[Thu Mar 04 15:50:05.194181 2021] [ssl:info] [pid 24140:tid 140137550751488] [client 10.65.65.199:59618] AH01964: Connection to child 322 established (server machine.xyz.com:443)
[Thu Mar 04 15:50:05.194424 2021] [ssl:debug] [pid 24140:tid 140137550751488] ssl_engine_kernel.c(2353): [client 10.65.65.199:59618] AH02043: SSL virtual host for servername machine.xyz.com found
[Thu Mar 04 15:50:05.334191 2021] [ssl:debug] [pid 24140:tid 140137550751488] ssl_engine_kernel.c(2236): [client 10.65.65.199:59618] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
[Thu Mar 04 15:50:05.376284 2021] [ssl:debug] [pid 24140:tid 140137550751488] ssl_engine_kernel.c(383): [client 10.65.65.199:59618] AH02034: Initial (No.1) HTTPS request received for child 322 (server machine.xyz.com:443)
[Thu Mar 04 15:50:05.376441 2021] [ssl:debug] [pid 24140:tid 140137550751488] ssl_engine_kernel.c(746): [client 10.65.65.199:59618] AH02255: Changed client verification type will force renegotiation
[Thu Mar 04 15:50:05.376454 2021] [ssl:info] [pid 24140:tid 140137550751488] [client 10.65.65.199:59618] AH02221: Requesting connection re-negotiation
[Thu Mar 04 15:50:05.376517 2021] [ssl:debug] [pid 24140:tid 140137550751488] ssl_engine_kernel.c(975): [client 10.65.65.199:59618] AH02260: Performing full renegotiation: complete handshake protocol (client does support secure renegotiation)
[Thu Mar 04 15:50:05.376617 2021] [ssl:info] [pid 24140:tid 140137550751488] [client 10.65.65.199:59618] AH02226: Awaiting re-negotiation handshake
[Thu Mar 04 15:50:05.498008 2021] [ssl:debug] [pid 24140:tid 140137550751488] ssl_engine_kernel.c(1741): [client 10.65.65.199:59618] AH02275: Certificate Verification, depth 1, CRL checking mode: none (0) [subject: CN=XYZ Co Server TEST CA 13,OU=PKI,O=XYZ Co AG,C=DE / issuer: CN=XYZ Co Group Root TEST CA 13,OU=PKI,O=XYZ Co AG,C=DE / serial: 04 / notbefore: Mar 18 00:00:00 2018 GMT / notafter: Oct 18 00:00:00 2026 GMT]
[Thu Mar 04 15:50:05.498078 2021] [ssl:info] [pid 24140:tid 140137550751488] [client 10.65.65.199:59618] AH02276: Certificate Verification: Error (20): unable to get local issuer certificate [subject: CN=XYZ Co Server TEST CA 13,OU=PKI,O=XYZ Co AG,C=DE / issuer: CN=XYZ Co Group Root TEST CA 13,OU=PKI,O=XYZ Co AG,C=DE / serial: 04 / notbefore: Mar 18 00:00:00 2018 GMT / notafter: Oct 18 00:00:00 2026 GMT]
[Thu Mar 04 15:50:05.498183 2021] [ssl:error] [pid 24140:tid 140137550751488] [client 10.65.65.199:59618] AH02261: Re-negotiation handshake failed
[Thu Mar 04 15:50:05.498223 2021] [ssl:error] [pid 24140:tid 140137550751488] SSL Library Error: error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed
[Thu Mar 04 15:50:05.498280 2021] [ssl:info] [pid 24140:tid 140137550751488] [client 10.65.65.199:59618] AH02008: SSL library error 1 in handshake (server machine.xyz.com:443)
[Thu Mar 04 15:50:05.498299 2021] [ssl:info] [pid 24140:tid 140137550751488] SSL Library Error: error:140800FF:SSL routines:ssl3_accept:unknown state
[Thu Mar 04 15:50:05.498314 2021] [ssl:info] [pid 24140:tid 140137550751488] [client 10.65.65.199:59618] AH01998: Connection closed to child 322 with abortive shutdown (server machine.xyz.com:443)

Apache Version

$ httpd -l
Compiled in modules:
  core.c
  mod_so.c
  http_core.c
  event.c
$ httpd -v
Server version: Apache/2.4.46 (Unix)
Server built:   Dec 23 2020 08:36:14

Apache Configuration

<VirtualHost _default_:8447>
    ServerAdmin some-support@xyz.com
    ServerName machine.xyz.com
    ServerAlias machine.xyz.com


    DocumentRoot /applications/apache2/htdocs_dev


    SSLEngine on
## Strengthen Ciphers
# Enable TLSv1.2, disable SSLv3.0, TLSv1.0 and TLSv1.1
SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1


# Enable modern TLS cipher suites
SSLCipherSuite          ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256


# The order of cipher suites matters
SSLHonorCipherOrder     on


# Disable TLS compression
SSLCompression          on


# Necessary for Perfect Forward Secrecy (PFS)
SSLSessionTickets       off
        SSLCertificateFile /applications//pki/tls/certs/machine.xyz.com_8447.crt
        SSLCertificateKeyFile /applications/pki/tls/certs/machine.xyz.com_8447.key
        SSLCertificateChainFile /applications/pki/tls/certs/xyz-sslca-certificate.crt


<Location /data-requests/secure>
        SSLVerifyClient require
        #SSLVerifyClient optional_no_ca
        SSLVerifyDepth 10
        SSLOptions +StdEnvVars +ExportCertData
</Location>


</VirtualHost>

score 3 · Answer 1 · answered Mar 04 '21 at 16:31

It is possible that the certificates you're using are invalid for mutual authentication. You need to ensure the client certificate has the ClientAuth extended key usage. https://en.wikipedia.org/wiki/X.509 talks about the extended key usage parameter on certificates. Similarly the server must have the ServerAuth extended key usage.

Outside of Apache2/curl you can use the openssl verify command discussed in Verify a certificate chain using openssl verify

In your current configuration of using SSLVerifyClient optional_no_ca then someone could mint a certificate with a matching CN from a different CA. The point of require instead of optional_no_ca is that it checks that the certificate comes from a trusted CA. In this case you need to make sure that trusted CA is referenced in either SSLCACertificateFile or SSLCACertificatePath.

1: https://httpd.apache.org/docs/current/mod/mod_ssl.html#sslcacertificatepath

Hi Jokajak - many thanks for your prompt response. It was very much appreciated. I was able to use openssl verify to confirm that l have the correct kind of keys. And the additional Apache2 variables. It allowed me to find the issue. Let me post in now. — Sukhster, Mar 04 '21 at 16:51

score 2 · Answer 2 · answered Mar 04 '21 at 17:04

Hi Jojajak - many thanks for the prompt response. It allowed me to fix my issue.

Your ONE post has increased my knowledge of this area ten-fold.

Verify a complete certificate

openssl verify -CAfile 'XYZ Company Root TEST CA 13.crt' -untrusted 'XYZ Company TEST CA 13.crt' client-machine.xyz.com_Testing-DoD-Application.crt
client-machine.xyz.com_Testing-DoD-Application.crt: OK

Confirm extended key usage

openssl x509 -in client-machine.xyz.com_Testing-DoD-Application.crt -purpose -noout -text
Certificate purposes:
SSL client : Yes
SSL client CA : No
SSL server : No
SSL server CA : No
Netscape SSL server : No
Netscape SSL server CA : No
S/MIME signing : No
S/MIME signing CA : No
S/MIME encryption : No
S/MIME encryption CA : No
CRL signing : No
CRL signing CA : No
Any Purpose : Yes
Any Purpose CA : Yes
OCSP helper : Yes
OCSP helper CA : No
Time Stamp signing : No
Time Stamp signing CA : No
. . . .

Issue was that l had not used SSLCACertificateFile in my Apache2 configuration. I replicated SSLCertificateChainFile to be SSLCACertificateFile and it worked.

    SSLCertificateFile /applications/pki/tls/certs/machine.xyz.com_8447.crt
    SSLCertificateKeyFile /applications/pki/tls/certs/machine.xyz.com_8447.key
    SSLCertificateChainFile /applications/pki/tls/certs/xyz-sslca-certificate.crt
## Below was the missing parameter. Duh!
    SSLCACertificateFile  /applications/pki/tls/certs/xyz-sslca-certificate.crt

Question closed, successfully.

Apache 2 Mutual TLS Authentication Only Working with "SSLClientVerify optional_no_ca"

Testing with Chrome

Apache Log

Testing Using Curl

Apache Log

Apache Version

Apache Configuration

2 Answers2