TLS peers can verify if a certificate was revoked by checking the CRL (very old and very poorly performing method with lots of shortcomings) or query the OCSP endpoint of the CA that issued the certificate.
However this design still has a shortcoming: what happens if for any reason the OCSP endpoint is unreachable (by accident or by anything caused by the evil people out there)? The outcome is a security risk - if the policy is to deny connection if the OCSP status cannot be checked, you risk to disserve.
Conversely, if the policy is that OCSP status check is a nice to have, there’s a risk that, if a revoked certificate has been stolen by the evils out there, they can just prevent your client to query the OCSP server and hijack the connection to a rogue TLS server managed by them that uses the stolen revoked certificate. To mitigate this you can set up OCSP stapling, which consists of prefetching OCSP responses and attaching them to the X.509 certificate.
In the "Apache HTTPd With Mutual TLS and OCSP Stapling" post we see not only how to configure an Apache server to provide a stapled certificate, but also how to set up mutual TLS authentication, seeing in action what happens when a certificate is revoked.
Overview
Setting up Apache HTTPd with TLS guarantees data confidentiality, since data on the wires are encrypted, but it does not provide any kind of authentication: to address this you can of course configure any kind of authentication mechanisms (such as basic auth, LDAP, Kerberos). One of the available ones is mutual-TLS - this mechanism requires the client to provide a client certificate that must be signed by a trusted Certificate Authority - authorization rules can then be applied by checking attributes on the certificate. Both server and client can make sure the peer’s certificate has not been revoked by querying the OCSP endpoint provided by the certificate.
But just relying on the CA’s OCSP responder has the availability shortcomings we just discussed, and can be bypassed by causing a denial of the OCSP service when running malicious attacks as described above.
One way to mitigate this is to set up OCSP stapling - when this is enabled, the TLS server pre-fetches OCSP replies from the OCSP server and staple them to its TLS certificate, sparing the client from having to contact the OCSP server.
The Lab
In this example we configure the "www.lab.carcano.corp" Apache virtual as follows:
- bind to port 443 (the IANA registered https port) and use TLS
- prefetch OCSP responses for its server certificate and staples it to the certificate provided to the client
- when a client try to access the "/restricted" HTTP path, it requires mutual TLS authentication, granting access only to clients providing a certificate belonging to the "Big Partner Foo SA" organization signed by the "carc i101l" Certificate Authority
- checks the OCSP status of the supplied client certificate
On his side, the client verifies the OCSP status of the Apache HTTPd server's certificate by directly checking the stapled OCSP response, without having to call the CA's OCSP endpoint.
Install Apache HTTPd With Mod_ssl
First and foremost, on the host you want to set up the web service, install Apache HTTPd with "mod_ssl" as follows:
dnf install -y httpd mod_ssl
It is of course necessary to add the firewall exception for accessing the https service:
firewall-cmd --add-service=https --permanent
firewall-cmd --reload
then create the directories where to put:
- the virtual hosts' configuration files ("/etc/httpd/vhosts.d")
- the virtual hosts' log files ("/var/log/httpd/vhosts")
- the trusted certificates used to validate the certificate's issuer when performing mutual-TLS authentication ("/etc/httpd/trusts")
mkdir -m 0755 /etc/httpd/vhosts.d /var/log/httpd/vhosts /etc/httpd/trusts
accordingly to the guidelines for a clean and scalable setup, configure the main Apache configuration file to look for additional configuration snippets files in the "/etc/httpd/vhosts.d" directory:
echo "IncludeOptional vhosts.d/*.conf" >> /etc/httpd/conf/httpd.conf
Configure The www.lab.carcano.corp Virtual Host
The "www.lab.carcano.corp" virtual host's root directory is "/var/www/vhosts/lab.carcano.corp/www" – create it as follows, along with the "restricted" subdirectory:
mkdir -m 0755 /var/www/vhosts \
/var/www/vhosts/lab.carcano.corp \
/var/www/vhosts/lab.carcano.corp/www \
/var/www/vhosts/lab.carcano.corp/www/restricted \
/var/log/httpd/vhosts/lab.carcano.corp
Configure where to store and the size of the OCSP stapling cache - add to the "/etc/httpd/conf.d/ssl.conf" file the following directive outside the definition of the default virtual host:
SSLStaplingCache shmcb:/var/run/httpd/ocsp(128000)
where:
- 128000 is the size you want to assign to the OCSP stapling cache
- "/var/run/httpd/ocsp" is the path to the OCSP cache file
We can now add the "www.lab.carcano.corp" virtual host include file - create the "/etc/httpd/vhosts.d/www.lab.carcano.corp.conf" file with the following contents:
<VirtualHost *:443>
ServerAdmin webmaster@lab.carcano.corp
ServerName lab.carcano.corp
ServerAlias www.lab.carcano.corp
DocumentRoot /var/www/vhosts/lab.carcano.corp/www
LogLevel notice
ErrorLog logs/vhosts/lab.carcano.corp/error.log
CustomLog logs/vhosts/lab.carcano.corp/access.log combined
<IfModule mod_ssl.c>
SSLEngine on
SSLCertificateFile /etc/pki/tls/certs/lab.carcano.corp.crt
SSLCertificateKeyFile /etc/pki/tls/private/lab.carcano.corp.key
SSLCertificateChainFile /etc/pki/ca-trust/source/anchors/carc_r100l.crt
SSLUseStapling on
# /etc/httpd/trusts/www.lab.carcano.corp.crt contains merged within the same file
# the certificates of each CA that is entitled to sign valid mutual-TLS certificates
# To create this file, for each certificate you put in the trust store, issue:
# openssl x509 -subject -in CAcertificatefile.crt
# and concatenate the output to the trustsore file using shell
# redirection ( >> )
SSLCACertificateFile /etc/httpd/trusts/www.lab.carcano.corp
SSLVerifyDepth 2
SSLOCSPEnable leaf
SSLOCSPUseRequestNonce off
</IfModule>
<Directory />
Options FollowSymLinks
AllowOverride Fileinfo
</Directory>
<Directory /var/www/vhosts/lab.carcano.corp/www/restricted>
Options Indexes FollowSymLinks
AllowOverride All
<IfModule mod_ssl.c>
SSLRequireSSL
SSLVerifyClient require
SSLOptions +StdEnvVars
<RequireAny>
# The full list of options is available at
# https://httpd.apache.org/docs/2.4/mod/mod_ssl.html#ssloptions
Require expr %{SSL_CLIENT_S_DN_O} == "Big Partner Foo SA"
</RequireAny>
</IfModule>
<IfModule !mod_ssl.c>
<RequireAny>
# The full list of options is available at
# https://httpd.apache.org/docs/2.4/mod/mod_ssl.html#ssloptions
Require all denied
</RequireAny>
</IfModule>
</Directory>
</VirtualHost>
the most interesting TLS related statement of the above configuration are:
- enabling the SSL engine setting the options for the whole virtual host (lines 9-25)
- configuring the server's certificate and key, along with the CA bundle with the certificate used to sign the server's certificate (lines 11-13)
- enable OCSP stapling (line 14)
- specify the trust store to use for mutual-TLS authentication (line 21)
- OCSP validation rules to apply to the client certificate (lines 22-24) - please note how here we relaxed the check a bit, disabling OCSP checks of the CA's certificate used to sign the client's certificate (line 23, value "leaf")
- require mutual-TLS authentication for accessing the "/restricted" web path (lines 34-36)
- restrict access to the "/restricted" web path only to clients providing a certificate belonging to the "Big Partner Foo SA" (The "O" part of the DN must be "O=Big Partner SA")
- prevent access to anybody if for any reason the SSL module was not loaded (lines 43-49)
Please mind that all of the above TLS settings refer just to the "www.lab.carcano.corp" Virtual Host.
Create A TrustStore
As we saw, the Mutual-TLS client's certificates are authenticated using the trust store specified by the "SSLCACertificateFile" directive.
To quickly identify the Certificate Authorities' certificates to put into the trust store, let's have a look to the "Big Partnert SA" "data-exchange-sv01" web service's certificate:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
0a:af:0f:42:cb:eb:38:1f:15:16:09:d2:39:ac:a9:e5:ea:8e:ca:fc
Signature Algorithm: ecdsa-with-SHA512
Issuer: C = CH, O = Carcano SA, CN = CARC I101L
Validity
Not Before: Feb 21 22:34:00 2024 GMT
Not After : Feb 20 22:34:00 2025 GMT
Subject: C = CH, O = Big Partner Foo SA, CN = data-exchange-svc01
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:aa:f2:65:c3:4d:8a:ab:c4:15:a8:9e:63:3b:55:
3a:16:25:48:a3:7b:b1:b4:2f:e3:b5:46:e7:20:17:
39:3c:4c:86:4f:f4:ab:2d:94:04:17:90:48:6b:a6:
ab:ab:8f:5a:47:45:9f:c9:69:99:26:ec:27:d2:7d:
31:19:ad:ff:bf
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Authority Key Identifier:
5C:A6:23:E7:26:5C:AB:EC:07:B0:74:6A:2C:A4:AA:BA:90:07:92:1B
Authority Information Access:
OCSP - URI:http://ocsp01.lab.carcano.corp:9401
CA Issuers - URI:http://ca01.lab.carcano.corp/carc_i101l/ca.crt
X509v3 CRL Distribution Points:
Full Name:
URI:http://ca01.lab.carcano.corp/carc_i101l/crl
X509v3 Subject Key Identifier:
9A:8A:C8:85:E4:12:56:6B:3B:5A:F1:CE:5F:76:DF:E8:75:36:E6:21
X509v3 Certificate Policies:
Policy: X509v3 Any Policy
CPS: http://ca01.lab.carcano.corp/carc_i101l/cps.html
Signature Algorithm: ecdsa-with-SHA512
Signature Value:
30:81:87:02:41:43:7d:51:9b:c4:ef:e8:7c:ff:99:95:95:83:
ef:b5:30:a7:4a:08:98:47:ae:bb:1e:d4:4b:df:6c:f9:03:55:
e1:35:7f:db:63:6f:b6:4e:aa:e1:8a:97:6b:91:d7:57:aa:21:
04:73:88:03:c8:5f:d1:de:d7:57:b0:10:ce:a4:22:de:02:42:
01:83:8e:5d:c6:5c:7b:f8:b1:60:c5:76:bf:cd:b4:7a:16:02:
8a:64:01:f3:88:b1:ba:2e:46:13:e8:96:08:4e:10:e7:d5:3e:
22:82:05:c8:63:99:07:d3:29:97:c5:07:33:f7:b1:8a:97:16:
bb:97:c9:46:ab:fd:31:f4:dc:ad:18:87
As claimed by the "Issuer" attribute, the client certificate is signed by the "carc_i101l" Intermediate CA.
Let's download it from the "CA Issuers" URL specified in the certificate's "Authority Information Access" X.509 extension by running the following statement:
wget -qO- http://ca01.lab.carcano.corp/carc_i101l/ca.crt | \
openssl x509 -subject >> /etc/httpd/trusts/www.lab.carcano.corp
since it is very likely that it is an Intermediate certificate, we need to inspect it to to guess the Root certificate used to sign it:
openssl x509 -in /etc/httpd/trusts/www.lab.carcano.corp -noout -text
the output is as follows:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
6e:d6:a9:50:55:4f:0d:05:e0:01:59:ca:b5:6f:59:84:56:79:60:ba
Signature Algorithm: ecdsa-with-SHA512
Issuer: C = CH, ST = Tessin, O = Carcano SA, CN = CARC R100L
Validity
Not Before: Feb 21 22:29:00 2024 GMT
Not After : Feb 20 22:29:00 2039 GMT
Subject: C = CH, O = Carcano SA, CN = CARC I101L
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (521 bit)
pub:
04:00:a1:0e:b0:44:98:55:1d:1c:d7:2e:bf:97:8b:
11:52:01:41:a0:55:7c:2c:ac:dc:6d:e7:c6:7c:54:
64:97:d7:65:06:86:fd:89:e0:b3:df:3b:f3:39:9d:
fc:73:30:4e:7d:bc:c1:c5:05:c0:5a:e3:f8:db:cf:
35:b6:f4:e3:e3:13:57:00:21:f6:98:91:52:ed:98:
66:59:a1:53:e9:74:c7:d0:98:c0:5b:19:e3:8a:98:
10:43:90:cc:94:7c:c3:63:2e:d1:e8:2b:d7:95:25:
d1:68:95:c4:b5:f8:5e:f0:e1:58:ca:41:a7:b5:7f:
8e:c1:c2:f8:c0:f3:85:33:98:69:ba:a6:c7
ASN1 OID: secp521r1
NIST CURVE: P-521
X509v3 extensions:
X509v3 Key Usage: critical
Certificate Sign, CRL Sign
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Key Identifier:
5C:A6:23:E7:26:5C:AB:EC:07:B0:74:6A:2C:A4:AA:BA:90:07:92:1B
X509v3 Authority Key Identifier:
47:0C:11:DD:74:C8:9B:B6:D7:C9:D3:9F:13:33:98:23:D8:9D:66:D1
Authority Information Access:
OCSP - URI:http://ocsp01.lab.carcano.corp:9400
CA Issuers - URI:http://ca01.lab.carcano.corp/carc_r100l/ca.crt
X509v3 CRL Distribution Points:
Full Name:
URI:http://ca01.lab.carcano.corp/carc_r100l/crl
X509v3 Certificate Policies:
Policy: X509v3 Any Policy
User Notice:
Explicit Text: CARC R100L Certification Practice Statement
CPS: http://ca01.lab.carcano.corp/carc_r100l/cps.html
Signature Algorithm: ecdsa-with-SHA512
Signature Value:
30:81:87:02:41:2c:c3:a9:7a:72:3a:10:8f:e4:09:cc:a0:e4:
06:75:47:49:95:04:5c:e9:51:52:07:d3:6c:93:88:8e:dc:7b:
b8:06:25:61:bc:55:a3:9a:aa:63:5e:70:cc:38:e9:bc:b0:62:
a4:74:6a:e6:2a:d3:79:66:bc:68:49:39:c6:24:6b:dd:02:42:
00:e5:39:8b:fc:bc:07:e0:d7:7d:75:24:f3:ea:44:80:e7:e0:
bb:0a:27:51:34:82:14:b7:16:79:89:72:69:cc:39:e8:7c:e1:
cb:95:9f:22:b5:0e:a0:03:13:d5:f1:b5:b5:33:f2:0f:41:98:
7a:0c:78:f1:05:ee:bc:8f:86:1f:d0:10
As you see the "carc_i101l" Intermediate CA's certificate is instead signed by the "carc_r100l" Root CA: we must download it too, again from the "CA Issuers" URL specified in the certificate's "Authority Information Access" X.509 extension, and append it to the same file by running the following statement:
wget -qO- http://ca01.lab.carcano.corp/carc_r100l/ca.crt | \
openssl x509 -subject >> /etc/httpd/trusts/www.lab.carcano.corp
Install And Configure Certmgr
We are still missing the TLS server's certificate used by the Apache HTTPd server itself – since we are going to manage its lifecycle using "certmgr", it is mandatory to install it as described in the "Installing" and "Basic Settings" paragraph of the "Cloudflare's Certmgr Tutorial – A Certmgr HowTo" post.
Retrieve The Authentication Key For Accessing The CFSSL CA
While enrolling certificates, certmgr connects to the Cloudflare's PKI and TLS Toolkit "multi-rootca" instance and claims it needs a certificate signed by the "carc i101l" CA.
Since that CA is configured to authenticate requests using a shared secret, it is necessary to provide that secret in the certmgr manifests.
To retrieve this secret, on the PKI host, run the following statement:
jq -r .auth_keys.default.key /etc/cfssl/tier01/carc_i101l/profiles/auth.json
the output of the above command is the value you have to set - in this example the output is:
aabbccddeeff11223344556677889900
Create The Manifest For Managing The Apache's Virtual Host Certificate
On the host running Apache HTTPd, set the value you got as "AUTH_KEY" environment variable and export it by running the following statement:
export AUTH_KEY=aabbccddeeff11223344556677889900
We can finally create the “www.lab.carcano.corp” certificate's manifest file – just run the following statement:
cat << EOF > /etc/certmgr/conf.d/www.lab.carcano.corp.json
{
"service": "httpd",
"action": "restart",
"request": {
"CN": "lab.carcano.corp",
"key": {
"algo": "ecdsa",
"size": 256
},
"hosts": [
"lab.carcano.corp",
"www.lab.carcano.corp",
"ftp.lab.carcano.corp"
],
"names": [
{
"C": "CH",
"ST": "Tessin",
"O": "Carcano SA"
}
]
},
"private_key": {
"path": "/etc/pki/tls/private/lab.carcano.corp.key",
"owner": "root",
"group": "apache",
"mode": "0640"
},
"certificate": {
"path": "/etc/pki/tls/certs/lab.carcano.corp.crt",
"owner": "root",
"group": "root",
"mode": "0644",
"key_usages": [
"signing",
"key encipherment",
"server auth"
]
},
"authority": {
"auth_key": "${AUTH_KEY}",
"label": "carc_i101l",
"profile": "server",
"root_ca": "/etc/pki/ca-trust/source/anchors/carc_r100l.crt"
}
}
EOF
we must of course add the "carc r100l" CA's certificate to the system-wide trust store:
wget -qO- http://ca01.lab.carcano.corp/carc_r100l/ca.crt | \
openssl x509 -subject > /etc/pki/ca-trust/source/anchors/carc_r100l.crt
update-ca-trust enable
update-ca-trust extract
this last step is necessary since certmgr itself, when submitting the CSR to the online Registration Authority, validates this endpoint's TLS certificate (signed by the "carc r100l" CA).
Enroll The Apache HTTPd's Virtual Host Server Certificate
We are ready to have certmgr enroll the certificate - just restart the "certmgr" service so to have it creating the certificate file and automatically restarting the Apache server:
systemctl restart certmgr
Testing The Setup
We have finally completed the setup, ... here begins the most interesting and funny part, that enables us to really learn if and how things are actually working.
As you may remember, TLS clients can automatically retrieve the certificates used for signing a leaf certificate from the URL claimed in the AIA X509v3 extension.
Unfortunately clients linked to the OpenSSL library, such as curl, do not download the CA's certificate from the URL specified in the AIA extension. For this reason, on the host we are using for testing the setup, it is necessary to add to the system-wide trust-store not only the "carc r100l" CA's certificate, but also the "carc i101l" CA's certificate. To do so, just issue the following statements:
wget -qO- http://ca01.lab.carcano.corp/carc_r100l/ca.crt | \
openssl x509 -subject > /etc/pki/ca-trust/source/anchors/carc_r100l.crt
wget -qO- http://ca01.lab.carcano.corp/carc_i101l/ca.crt | \
openssl x509 -subject > /etc/pki/ca-trust/source/anchors/carc_i101l.crt
update-ca-trust enable
update-ca-trust extract
Verify OCSP Stapling On The Apache HTTPd's Certificate
The first check is making sure Apache's HTTPd is actually stapling to it's TLS certificate the OCSP responses - this can be easily done by using the "openssl s_client" command line utility as follows:
openssl s_client -connect www.lab.carcano.corp:443 -tlsextdebug -status
The Apache's HTTPd answer must contain the stapled OCSP response, as by the first lines in the following snippet (I have cut only the relevant part):
depth=2 C = CH, ST = Tessin, O = Carcano SA, CN = CARC R100L
verify return:1
depth=1 C = CH, O = Carcano SA, CN = CARC I101L
verify return:1
depth=0 C = CH, ST = Tessin, O = Carcano SA, CN = www.lab.carcano.corp
verify return:1
OCSP response:
======================================
OCSP Response Data:
OCSP Response Status: successful (0x0)
Response Type: Basic OCSP Response
Version: 1 (0x0)
Responder Id: C = CH, O = Carcano SA, CN = CARC I101L OCSP
Produced At: Oct 13 16:50:00 2023 GMT
Responses:
Certificate ID:
Hash Algorithm: sha1
Issuer Name Hash: 86099856EDC298EB986CA38CBF79DB28B27C9598
Issuer Key Hash: E33E708DCA69A73D6102E9EF032EB7A0F8A378B4
Serial Number: 1315B0AFB200C223AEDA04D8E061F42EA84DD00B
Cert Status: good
This Update: Oct 13 16:00:00 2023 GMT
Next Update: Oct 17 16:00:00 2023 GMT
Signature Algorithm: ecdsa-with-SHA512
Signature Value:
30:81:87:02:41:05:d1:53:9c:04:4b:97:ab:21:1a:4b:64:5c:
cf:45:69:f5:46:96:31:92:77:ed:75:7c:75:79:72:30:e2:17:
4d:0c:56:96:91:0b:bf:cb:1c:f8:1b:3d:26:3b:03:f8:61:4f:
4a:78:a2:31:e3:68:d4:7a:38:ef:03:53:03:26:86:e2:02:42:
01:7a:72:00:6e:1c:c6:ae:28:d6:e4:fb:66:80:46:bd:b6:ed:
ff:3b:6b:cd:89:63:e1:dc:2a:c9:f0:17:e9:2e:ff:cd:3f:17:
ed:e5:17:eb:3c:e5:57:6b:0c:34:94:f8:1d:a1:b4:08:12:da:
3f:16:93:17:eb:1a:55:ec:a2:5f:d9:c2
if instead you get something like the below snippet:
depth=0 C = CH, ST = Tessin, O = Carcano SA, CN = www.lab.carcano.corp
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = CH, ST = Tessin, O = Carcano SA, CN = www.lab.carcano.corp
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 C = CH, ST = Tessin, O = Carcano SA, CN = www.lab.carcano.corp
verify return:1OCSP response:
======================================
OCSP Response Data:
OCSP Response Status: trylater (0x3)
======================================
it means the Apache HTTPd server had troubles contacting the CA's OCSP responder and so it was unable to staple a currently valid OCSP response.
This kind of problem can be easily identified also by monitoring log files - when the client connects, Apache tries to contact the CA's OCSP responder: if anything would go wrong, you'll find entries like the below one in the main SSL error log file ("/var/log/httpd/ssl_error_log"):
[Wed Mar 06 07:01:44.818500 2024] [ssl:error] [pid 10891:tid 10993] (111)Connection refused:
[client 10.211.55.185:42136] AH01974: could not connect to OCSP responder 'ocsp01.lab.carcano.corp:9401'
Apache logs the error also in the Virtual Host's specific error log file ("/var/log/httpd/vhosts/lab.carcano.corp/error.log"):
[Wed Mar 06 07:01:44.818560 2024] [ssl:error] [pid 10891:tid 10993]
AH01941: stapling_renew_response: responder error
Test Access To Public Documents
The next check is to make sure the server correctly provides public documents - or in other words, clients not providing client certificates get the public available data.
To test it, let's first create the index document in the Virtual Host's document root:
cat << EOF > /var/www/vhosts/lab.carcano.corp/www/index.html
<HTML>
<BODY>
<H1>WWW.LAB.CARCANO.CORP</H1>
</BODY>
</HTML>
EOF
Let's now try getting the document root of the "https://www.lab.carcano.corp" Virtual Hosts.
Since it is the index page, it is not necessary to specify it in the URL - just type:
curl https://www.lab.carcano.corp/
The outcome must be as follows:
<HTML>
<BODY>
<H1>WWW.LAB.carcano.corp</H1>
</BODY>
</HTML>
Test Access To Access Restricted Documents
Since we are also providing access restricted contents, we need to make sure restrictions are actually enforced.
The current rule restricts access to the "/restricted" web path only to clients providing a certificate belonging to the "Big Partner Foo SA" (The "O" part of the DN must be "O=Big Partner SA").
To test it, let's first create the "sensitive.txt" document in the Virtual Host's "restricted" directory:
cat << EOF > /var/www/vhosts/lab.carcano.corp/www/restricted/sensitive.txt
This is a sensitive document
EOF
Let's first test an anonymous access to the access restricted document - that means the client does not provide an X.509 client certificate:
curl https://www.lab.carcano.corp/restricted/sensitive.txt
The expected outcome is the following error:
curl: (56) OpenSSL SSL_read: error:0A00045C:SSL routines::tlsv13 alert certificate
required, errno 0
So the server correctly answers us that to access the requested URL it is necessary to provide a client certificate to attempt mutual-TLS authentication.
Let's re-try, but this time providing the "~/foo-data-exchange-svc01.crt" certificate and the related "~/foo-data-exchange-svc01.key" private key - the ones of the "Big Partnert SA" "data-exchange-sv01" identity :
curl--cert ~/foo-data-exchange-svc01.crt --key ~/foo-data-exchange-svc01.key \
https://www.lab.carcano.corp/restricted/sensitive.txt
when requested, type the password to unlock the private key and hit enter.
This time it must work, and we must get the document:
This is a sensitive document
If instead you get the following error message:
curl: (56) OpenSSL SSL_read: error:0A000410:SSL routines::sslv3
alert handshake failure, errno 0
it can mean that
- the OCSP response for this client certificate has not been generated yet, and so Apache was not able to verify it. Server side it logged the message "AH02263: Re-negotiation handshake failed: Client certificate missing" in the "/var/log/httpd/vhosts/lab.carcano.corp/error.log" log file. In such a scenario you just must wait until the OCSP response for this client certificate gets generated before retrying to connect to the Apache HTTPd server
- the OCSP responder of the CA that signed the client certificate is unreachable. Server side it is logged the message "AH02262: Re-negotiation handshake failed: Client verification failed" in the "/var/log/httpd/vhosts/lab.carcano.corp/error.log" log file and the message "AH01974: could not connect to OCSP responder 'ocsp01.lab.carcano.corp:9401" in the "/var/log/httpd/ssl_error_log" log file.
- it is not the case for our lab, but the above message can be caused also by the client certificate signed by CA not trusted by the Apache HTTPd server's Virtual Host.
Attempting Mutual-TLS With The Revoked Certificate
Since we set up everything to work with OCSP, it is also necessary to test the behavior with revoked certificates.
Let's start by revoking the client's "Big Partnert SA" "data-exchange-sv01"'s certificate using CFSSL - and wait the time to have OCSP responses re-generated.
After revoking it, run a mutual-TLS authenticated connection attempt again to see if the Apache web server denies us access because of the revocation of the certificate.
As we did before, just run:
curl --cert ~/foo-data-exchange-svc01.crt --key ~/foo-data-exchange-svc01.key \
https://www.lab.carcano.corp/restricted/sensitive.txt
when requested, type the password to unlock the private key and hit enter.
The expected behaviour is Apache denying access and curl showing an error message like that:
curl: (56) OpenSSL SSL_read: error:0A000414:SSL routines::sslv3 alert certificate revoked, errno 0
server side, Apache logs the following error message in the "/var/log/httpd/ssl_error_log" file:
Tue Oct 17 06:44:20.871328 2023] [ssl:error] [pid 3833:tid 3876] [client 127.0.0.1:56190] AH03239: OCSP validation completed, certificate status: revoked (1, 4) [subject: CN=data-exchange-svc01,O=Big Partner Foo SA,C=CH / issuer: CN=CARC I101L,O=Carcano SA,C=CH / serial: 4D1ED13DE091CE3763ED1F9BE99A35A269598BE3 / notbefore: Oct 13 16:08:00 2023 GMT / notafter: Oct 12 16:08:00 2024 GMT]
[Tue Oct 17 06:44:20.871365 2023] [ssl:error] [pid 3833:tid 3876] [client 127.0.0.1:56190] AH02039: Certificate Verification: Error (23): certificate revoked
and the following error message in the "/var/log/httpd/vhosts/lab.carcano.corp/error.log" file:
[Tue Oct 17 07:13:29.340549 2023] [ssl:error] [pid 3835:tid 4029] [client 127.0.0.1:33386] AH02262: Re-negotiation handshake failed: Client verification failed
It also log an access denied (HTTP status 403) in the "/var/log/httpd/vhosts/lab.carcano.corp/access.log" file:
10.4.5.88 - - [17/Oct/2023:06:48:42 +0200] "GET /restricted/sensitive.txt HTTP/1.1" 403 199 "-" "curl/7.76.1"
Footnotes
Here it ends the "Apache HTTPd With Mutual TLS and OCSP Stapling" post, our workshop on strong security on Apache working with mutual-TLS and with OCSP stapling - I hope you understood how all these security mechanisms play together and enable you to configure a very strong set up.
The only requirement to run it is of course running a Cloudflare's PKI and TLS toolkit based PKI: if you want to quickly and easily learn how to set it up and run it, I hope you can enjoy my book "A Full Featured PKI With Cloudflare's PKI and TLS Toolkit" available both as ebook and paperback.
Pedro says:
Hi Carcano
I have been doing internal root CAs via Openssl with ocsp stapling and
required user certificates for quite some time.
My root CA cert signs every certs csr there is to sign.
My case certs for vhosts on Linux and certs for Exchange.
I do not use MS Cert Auth. A bloody dawn mess by the way.
Anyway GPO work good providing user certs for the user.
User certs both in Apache for our vhosts and also for Exchange servers where access
to webmail requires user certificates.
Exchange web mail is 2 edge sword.
IIS have web sites Frontend and Backend where requiring user certs is not possible
since breaks internal Ofiice outlook clients.
So I needed to create an adittional IIS web site (OWA-FrontEnd) outside Frontend and Backend where
then I could activate required user certs.
With the additional reasoning that than I could restrict IP acces to only internal networks
for Frontend and Backend web site and only external acces to new web site via IIS “IP addr and domain restrictions”
Unfortunatly contrary to Apache where we can regexp test for the fields in the user cert, in IIS
the test is only presenting the user certs signed from the internal CA, there is no way to test for fields.
At least as far as I known.
User certs today are quite easy to add on mobile androids.
So everybody as far as they can should be using user certs as a strict security rule.
Because of the undeterminate web browsers out there I usually opt for RSA.
I had problems with the newer Elliptic curves on some client browsers
Openssl is quite difficult.
And confusing since there is always several ways to do things.
And if you mess up some fields on certs, after signing there is no way out
but revoke, update crls, resign again, restart ocspd since it reads index.
What a trouble.
Only 4 books.
Oreiily frst one Viega and friends
Ivan Ristic Feisty Duck
Packt Alexei Khlebnikov
TLS MAstery Michael Lucas
I do not known of any more
I have dozen of files where I document the work i have been doing for years.
They are my rescue because if I do not work on it for a while is a problem to remember.
If to much time not working I have to read them again.
To be honest I still do not believe much on automatic resign on near end dates.
Outside very simple setups is difficult, openssl too complex to something
automatic, shell scripts or whatever.
May be some crazy fellow comes by with an Ansible solutions for the problem.
Marco Antonio Carcano says:
Hi Pedro. For whatever is related to MS software’s shortcomings, my experience is very outdated, since I’ve been working only on Linux for the last 15 years, so I can just trust you :O) .
I agree, given how easy is installing certificates on Android, user’s certificates are a very good authentication mechanisms: I saw their usage in financial applications: the only thing to be wary is to use a private CA and implement pinning of the Issuing CA, so to avoid a public CA to issue a rogue certificate by mistake – it’s a very rare mistake, … but it can happen, and indeed it happened.
You are right: some (often old) browsers may have problems with EC certificates – a workaround is to provide both (AFAIK Apache should support repeating the multiple SSLCertificateFile and SSLCertificateKeyFile to address this specific use case).
OpenSSL is hard, … that’s why I tried to help by also writing a post on it. As for automatic enrollment of existing certificates right before their expirations, … the automation is certainly helpful, but don’t forget to monitor: you can periodically querying the endpoints where they are installed, … just to make sure the automation that is taking care of renewals was not on strike when it was supposed to renew them. The cfss-scan command line utility can be a good brick to use in automation scripts for this specific use case.
Cheers