Apache HTTPd With Mutual TLS and OCSP Stapling

TLS peers can verify if a certificate was revoked by checking the CRL (very old and very poorly performing method with lots of shortcomings) or query the OCSP endpoint of the CA that issued the certificate.

However this design still has a shortcoming: what happens if for any reason the OCSP endpoint is unreachable (by accident or by anything caused by the evil people out there)? The outcome is a security risk - if the policy is to deny connection if the OCSP status cannot be checked, you risk to disserve.

Conversely, if the policy is that OCSP status check is a nice to have, there’s a risk that, if a revoked certificate has been stolen by the evils out there, they can just prevent your client to query the OCSP server and hijack the connection to a rogue TLS server managed by them that uses the stolen revoked certificate. To mitigate this you can set up OCSP stapling, which consists of prefetching OCSP responses and attaching them to the X.509 certificate.

In the "Apache HTTPd With Mutual TLS and OCSP Stapling" post we see not only how to configure an Apache server to provide a stapled certificate, but also how to set up mutual TLS authentication, seeing in action what happens when a certificate is revoked.

This post is a sample from the book "A Full Featured PKI With Cloudflare's PKI and TLS Toolkit": if you are interested into a quick deep dive on this topic and be able to quickly set up a full featured nearly real-life lab with it, here is the link to the ebook and here the link to the paperback.

Overview

Setting up Apache HTTPd with TLS guarantees data confidentiality, since data on the wires are encrypted, but it does not provide any kind of authentication: to address this you can of course configure any kind of authentication mechanisms (such as basic auth, LDAP, Kerberos). One of the available ones is mutual-TLS - this mechanism requires the client to provide a client certificate that must be signed by a trusted Certificate Authority - authorization rules can then be applied by checking attributes on the certificate. Both server and client can make sure the peer’s certificate has not been revoked by querying the OCSP endpoint provided by the certificate.

But just relying on the CA’s OCSP responder has the availability shortcomings we just discussed, and can be bypassed by causing a denial of the OCSP service when running malicious attacks as described above.

One way to mitigate this is to set up OCSP stapling - when this is enabled, the TLS server pre-fetches OCSP replies from the OCSP server and staple them to its TLS certificate, sparing the client from having to contact the OCSP server.

The Lab

In this example we configure the "www.lab.carcano.corp" Apache virtual as follows:

• bind to port 443 (the IANA registered https port) and use TLS
• prefetch OCSP responses for its server certificate and staples it to the certificate provided to the client
• when a client try to access the "/restricted" HTTP path, it requires mutual TLS authentication, granting access only to clients providing a certificate belonging to the "Big Partner Foo SA" organization signed by the "carc i101l" Certificate Authority
• checks the OCSP status of the supplied client certificate

On his side, the client verifies the OCSP status of the Apache HTTPd server's certificate by directly checking the stapled OCSP response, without having to call the CA's OCSP endpoint.

The server's certificate is managed by a certmgr instance running on the same host of the Apache HTTPd server - this means that installing certmgr is a mandatory requisite to go on with this post. If you need guidelines, have a look at the "Installing" and "Basic Settings" paragraph of the "Cloudflare's Certmgr Tutorial – A Certmgr HowTo" post.

In this post we are using Apache 2.4: please mind that other Apache versions may have a slightly different configuration syntax.

Install Apache HTTPd With Mod_ssl

First and foremost, on the host you want to set up the web service, install Apache HTTPd with "mod_ssl" as follows:

dnf install -y httpd mod_ssl

It is of course necessary to add the firewall exception for accessing the https service:

firewall-cmd --add-service=https --permanent
firewall-cmd --reload

then create the directories where to put:

• the virtual hosts' configuration files ("/etc/httpd/vhosts.d")
• the virtual hosts' log files ("/var/log/httpd/vhosts")
• the trusted certificates used to validate the certificate's issuer when performing mutual-TLS authentication ("/etc/httpd/trusts")

mkdir -m 0755 /etc/httpd/vhosts.d /var/log/httpd/vhosts /etc/httpd/trusts 

accordingly to the guidelines for a clean and scalable setup, configure the main Apache configuration file to look for additional configuration snippets files in the "/etc/httpd/vhosts.d" directory:

 echo "IncludeOptional vhosts.d/*.conf" >> /etc/httpd/conf/httpd.conf

Configure The www.lab.carcano.corp Virtual Host

The "www.lab.carcano.corp" virtual host's root directory is "/var/www/vhosts/lab.carcano.corp/www" – create it as follows, along with the "restricted" subdirectory:

mkdir -m 0755 /var/www/vhosts \
/var/www/vhosts/lab.carcano.corp \
/var/www/vhosts/lab.carcano.corp/www \
/var/www/vhosts/lab.carcano.corp/www/restricted \
/var/log/httpd/vhosts/lab.carcano.corp

Configure where to store and the size of the OCSP stapling cache - add to the "/etc/httpd/conf.d/ssl.conf" file the following directive outside the definition of the default virtual host:

SSLStaplingCache shmcb:/var/run/httpd/ocsp(128000)

where:

• 128000 is the size you want to assign to the OCSP stapling cache
• "/var/run/httpd/ocsp" is the path to the OCSP cache file

We can now add the "www.lab.carcano.corp" virtual host include file - create the "/etc/httpd/vhosts.d/www.lab.carcano.corp.conf" file with the following contents:

<VirtualHost *:443>
  ServerAdmin webmaster@lab.carcano.corp
  ServerName lab.carcano.corp
  ServerAlias www.lab.carcano.corp
  DocumentRoot /var/www/vhosts/lab.carcano.corp/www
  LogLevel notice
  ErrorLog logs/vhosts/lab.carcano.corp/error.log
  CustomLog logs/vhosts/lab.carcano.corp/access.log combined
  <IfModule mod_ssl.c>
    SSLEngine on
    SSLCertificateFile    /etc/pki/tls/certs/lab.carcano.corp.crt
    SSLCertificateKeyFile /etc/pki/tls/private/lab.carcano.corp.key
    SSLCertificateChainFile /etc/pki/ca-trust/source/anchors/carc_r100l.crt
    SSLUseStapling on
    # /etc/httpd/trusts/www.lab.carcano.corp.crt contains merged within the same file
    # the certificates of each CA that is entitled to sign valid mutual-TLS certificates
    # To create this file, for each certificate you put in the trust store, issue:
    # openssl x509 -subject -in CAcertificatefile.crt
    # and concatenate the output to the trustsore file using shell
    # redirection ( >> )
    SSLCACertificateFile /etc/httpd/trusts/www.lab.carcano.corp
    SSLVerifyDepth 2
    SSLOCSPEnable leaf
    SSLOCSPUseRequestNonce off
  </IfModule>
  <Directory />
    Options FollowSymLinks
    AllowOverride Fileinfo
  </Directory>
  <Directory /var/www/vhosts/lab.carcano.corp/www/restricted>
    Options Indexes FollowSymLinks
    AllowOverride All
    <IfModule mod_ssl.c>
      SSLRequireSSL
      SSLVerifyClient require
      SSLOptions +StdEnvVars
      <RequireAny>
        # The full list of options is available at
        # https://httpd.apache.org/docs/2.4/mod/mod_ssl.html#ssloptions
        Require expr %{SSL_CLIENT_S_DN_O} == "Big Partner Foo SA"
      </RequireAny>
    </IfModule>
    <IfModule !mod_ssl.c>
      <RequireAny>
        # The full list of options is available at
        # https://httpd.apache.org/docs/2.4/mod/mod_ssl.html#ssloptions
        Require all denied
      </RequireAny>
    </IfModule>
  </Directory>
</VirtualHost>

the most interesting TLS related statement of the above configuration are:

• enabling the SSL engine setting the options for the whole virtual host (lines 9-25)
• configuring the server's certificate and key, along with the CA bundle with the certificate used to sign the server's certificate (lines 11-13)
• enable OCSP stapling (line 14)
• specify the trust store to use for mutual-TLS authentication (line 21)
• OCSP validation rules to apply to the client certificate (lines 22-24) - please note how here we relaxed the check a bit, disabling OCSP checks of the CA's certificate used to sign the client's certificate (line 23, value "leaf")
• require mutual-TLS authentication for accessing the "/restricted" web path (lines 34-36)
• restrict access to the "/restricted" web path only to clients providing a certificate belonging to the "Big Partner Foo SA" (The "O" part of the DN must be "O=Big Partner SA")
• prevent access to anybody if for any reason the SSL module was not loaded (lines 43-49)

Please mind that all of the above TLS settings refer just to the "www.lab.carcano.corp" Virtual Host.

Remember to add an entry for “www.lab.carcano.corp” in your DNS or in the "/etc/hosts" file of each host you are using in your lab.

Create A TrustStore

As we saw, the Mutual-TLS client's certificates are authenticated using the trust store specified by the "SSLCACertificateFile" directive.

To quickly identify the Certificate Authorities' certificates to put into the trust store, let's have a look to the "Big Partnert SA" "data-exchange-sv01" web service's certificate:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0a:af:0f:42:cb:eb:38:1f:15:16:09:d2:39:ac:a9:e5:ea:8e:ca:fc
        Signature Algorithm: ecdsa-with-SHA512
        Issuer: C = CH, O = Carcano SA, CN = CARC I101L
        Validity
            Not Before: Feb 21 22:34:00 2024 GMT
            Not After : Feb 20 22:34:00 2025 GMT
        Subject: C = CH, O = Big Partner Foo SA, CN = data-exchange-svc01
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:aa:f2:65:c3:4d:8a:ab:c4:15:a8:9e:63:3b:55:
                    3a:16:25:48:a3:7b:b1:b4:2f:e3:b5:46:e7:20:17:
                    39:3c:4c:86:4f:f4:ab:2d:94:04:17:90:48:6b:a6:
                    ab:ab:8f:5a:47:45:9f:c9:69:99:26:ec:27:d2:7d:
                    31:19:ad:ff:bf
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier: 
                5C:A6:23:E7:26:5C:AB:EC:07:B0:74:6A:2C:A4:AA:BA:90:07:92:1B
            Authority Information Access: 
                OCSP - URI:http://ocsp01.lab.carcano.corp:9401
                CA Issuers - URI:http://ca01.lab.carcano.corp/carc_i101l/ca.crt
            X509v3 CRL Distribution Points: 
                Full Name:
                  URI:http://ca01.lab.carcano.corp/carc_i101l/crl
            X509v3 Subject Key Identifier: 
                9A:8A:C8:85:E4:12:56:6B:3B:5A:F1:CE:5F:76:DF:E8:75:36:E6:21
            X509v3 Certificate Policies: 
                Policy: X509v3 Any Policy
                  CPS: http://ca01.lab.carcano.corp/carc_i101l/cps.html
    Signature Algorithm: ecdsa-with-SHA512
    Signature Value:
        30:81:87:02:41:43:7d:51:9b:c4:ef:e8:7c:ff:99:95:95:83:
        ef:b5:30:a7:4a:08:98:47:ae:bb:1e:d4:4b:df:6c:f9:03:55:
        e1:35:7f:db:63:6f:b6:4e:aa:e1:8a:97:6b:91:d7:57:aa:21:
        04:73:88:03:c8:5f:d1:de:d7:57:b0:10:ce:a4:22:de:02:42:
        01:83:8e:5d:c6:5c:7b:f8:b1:60:c5:76:bf:cd:b4:7a:16:02:
        8a:64:01:f3:88:b1:ba:2e:46:13:e8:96:08:4e:10:e7:d5:3e:
        22:82:05:c8:63:99:07:d3:29:97:c5:07:33:f7:b1:8a:97:16:
        bb:97:c9:46:ab:fd:31:f4:dc:ad:18:87 

As claimed by the "Issuer" attribute, the client certificate is signed by the "carc_i101l" Intermediate CA.

Let's download it from the "CA Issuers" URL specified in the certificate's "Authority Information Access" X.509 extension by running the following statement:

wget -qO- http://ca01.lab.carcano.corp/carc_i101l/ca.crt | \
 openssl x509 -subject  >> /etc/httpd/trusts/www.lab.carcano.corp

since it is very likely that it is an Intermediate certificate, we need to inspect it to to guess the Root certificate used to sign it:

openssl x509 -in /etc/httpd/trusts/www.lab.carcano.corp -noout -text

the output is as follows:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            6e:d6:a9:50:55:4f:0d:05:e0:01:59:ca:b5:6f:59:84:56:79:60:ba
        Signature Algorithm: ecdsa-with-SHA512
        Issuer: C = CH, ST = Tessin, O = Carcano SA, CN = CARC R100L
        Validity
            Not Before: Feb 21 22:29:00 2024 GMT
            Not After : Feb 20 22:29:00 2039 GMT
        Subject: C = CH, O = Carcano SA, CN = CARC I101L
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (521 bit)
                pub:
                    04:00:a1:0e:b0:44:98:55:1d:1c:d7:2e:bf:97:8b:
                    11:52:01:41:a0:55:7c:2c:ac:dc:6d:e7:c6:7c:54:
                    64:97:d7:65:06:86:fd:89:e0:b3:df:3b:f3:39:9d:
                    fc:73:30:4e:7d:bc:c1:c5:05:c0:5a:e3:f8:db:cf:
                    35:b6:f4:e3:e3:13:57:00:21:f6:98:91:52:ed:98:
                    66:59:a1:53:e9:74:c7:d0:98:c0:5b:19:e3:8a:98:
                    10:43:90:cc:94:7c:c3:63:2e:d1:e8:2b:d7:95:25:
                    d1:68:95:c4:b5:f8:5e:f0:e1:58:ca:41:a7:b5:7f:
                    8e:c1:c2:f8:c0:f3:85:33:98:69:ba:a6:c7
                ASN1 OID: secp521r1
                NIST CURVE: P-521
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier: 
                5C:A6:23:E7:26:5C:AB:EC:07:B0:74:6A:2C:A4:AA:BA:90:07:92:1B
            X509v3 Authority Key Identifier: 
                47:0C:11:DD:74:C8:9B:B6:D7:C9:D3:9F:13:33:98:23:D8:9D:66:D1
            Authority Information Access: 
                OCSP - URI:http://ocsp01.lab.carcano.corp:9400
                CA Issuers - URI:http://ca01.lab.carcano.corp/carc_r100l/ca.crt
            X509v3 CRL Distribution Points: 
                Full Name:
                  URI:http://ca01.lab.carcano.corp/carc_r100l/crl
            X509v3 Certificate Policies: 
                Policy: X509v3 Any Policy
                  User Notice:
                    Explicit Text: CARC R100L Certification Practice Statement
                  CPS: http://ca01.lab.carcano.corp/carc_r100l/cps.html
    Signature Algorithm: ecdsa-with-SHA512
    Signature Value:
        30:81:87:02:41:2c:c3:a9:7a:72:3a:10:8f:e4:09:cc:a0:e4:
        06:75:47:49:95:04:5c:e9:51:52:07:d3:6c:93:88:8e:dc:7b:
        b8:06:25:61:bc:55:a3:9a:aa:63:5e:70:cc:38:e9:bc:b0:62:
        a4:74:6a:e6:2a:d3:79:66:bc:68:49:39:c6:24:6b:dd:02:42:
        00:e5:39:8b:fc:bc:07:e0:d7:7d:75:24:f3:ea:44:80:e7:e0:
        bb:0a:27:51:34:82:14:b7:16:79:89:72:69:cc:39:e8:7c:e1:
        cb:95:9f:22:b5:0e:a0:03:13:d5:f1:b5:b5:33:f2:0f:41:98:
        7a:0c:78:f1:05:ee:bc:8f:86:1f:d0:10

As you see the "carc_i101l" Intermediate CA's certificate is instead signed by the "carc_r100l" Root CA: we must download it too, again from the "CA Issuers" URL specified in the certificate's "Authority Information Access" X.509 extension, and append it to the same file by running the following statement:

 wget -qO- http://ca01.lab.carcano.corp/carc_r100l/ca.crt | \
 openssl x509 -subject  >> /etc/httpd/trusts/www.lab.carcano.corp

Install And Configure Certmgr

We are still missing the TLS server's certificate used by the Apache HTTPd server itself – since we are going to manage its lifecycle using "certmgr", it is mandatory to install it as described in the "Installing" and "Basic Settings" paragraph of the "Cloudflare's Certmgr Tutorial – A Certmgr HowTo" post.

Retrieve The Authentication Key For Accessing The CFSSL CA

While enrolling certificates, certmgr connects to the Cloudflare's PKI and TLS Toolkit "multi-rootca" instance and claims it needs a certificate signed by the "carc i101l" CA.

Since that CA is configured to authenticate requests using a shared secret, it is necessary to provide that secret in the certmgr manifests.

To retrieve this secret, on the PKI host, run the following statement:

 jq -r .auth_keys.default.key /etc/cfssl/tier01/carc_i101l/profiles/auth.json

the output of the above command is the value you have to set - in this example the output is:

aabbccddeeff11223344556677889900

The above one is the only step to run on the PKI host: the next steps must be run on the host where we installed Apache with mod_ssl.

Create The Manifest For Managing The Apache's Virtual Host Certificate

On the host running Apache HTTPd, set the value you got as "AUTH_KEY" environment variable and export it by running the following statement:

export AUTH_KEY=aabbccddeeff11223344556677889900

We can finally create the “www.lab.carcano.corp” certificate's manifest file – just run the following statement:

cat << EOF > /etc/certmgr/conf.d/www.lab.carcano.corp.json
{
    "service": "httpd",
    "action": "restart",
    "request": {
        "CN": "lab.carcano.corp",
        "key": {
            "algo": "ecdsa",
            "size": 256
        },
        "hosts": [
            "lab.carcano.corp",
            "www.lab.carcano.corp",
            "ftp.lab.carcano.corp"
        ],
        "names": [
            {
                "C": "CH",
                "ST": "Tessin",
                "O": "Carcano SA"
            }
        ] 
    },
    "private_key": {
        "path": "/etc/pki/tls/private/lab.carcano.corp.key",
        "owner": "root",
        "group": "apache",
        "mode": "0640"
    },
    "certificate": {
        "path": "/etc/pki/tls/certs/lab.carcano.corp.crt",
        "owner": "root",
        "group": "root",
        "mode": "0644",
        "key_usages": [
            "signing",
            "key encipherment",
            "server auth"
        ]
    },
    "authority": {
        "auth_key": "${AUTH_KEY}",
        "label": "carc_i101l",
        "profile": "server",
        "root_ca": "/etc/pki/ca-trust/source/anchors/carc_r100l.crt"
    }
}
EOF

we must of course add the "carc r100l" CA's certificate to the system-wide trust store:

wget -qO- http://ca01.lab.carcano.corp/carc_r100l/ca.crt |  \
openssl x509 -subject  > /etc/pki/ca-trust/source/anchors/carc_r100l.crt
update-ca-trust enable
update-ca-trust extract

this last step is necessary since certmgr itself, when submitting the CSR to the online Registration Authority, validates this endpoint's TLS certificate (signed by the "carc r100l" CA).

Enroll The Apache HTTPd's Virtual Host Server Certificate

We are ready to have certmgr enroll the certificate - just restart the "certmgr" service so to have it creating the certificate file and automatically restarting the Apache server:

systemctl restart certmgr

Testing The Setup

We have finally completed the setup, ... here begins the most interesting and funny part, that enables us to really learn if and how things are actually working.

As you may remember, TLS clients can automatically retrieve the certificates used for signing a leaf certificate from the URL claimed in the AIA X509v3 extension.

Unfortunately clients linked to the OpenSSL library, such as curl, do not download the CA's certificate from the URL specified in the AIA extension. For this reason, on the host we are using for testing the setup, it is necessary to add to the system-wide trust-store not only the "carc r100l" CA's certificate, but also the "carc i101l" CA's certificate. To do so, just issue the following statements:

wget -qO- http://ca01.lab.carcano.corp/carc_r100l/ca.crt | \ 
openssl x509 -subject > /etc/pki/ca-trust/source/anchors/carc_r100l.crt
wget -qO- http://ca01.lab.carcano.corp/carc_i101l/ca.crt | \
openssl x509 -subject > /etc/pki/ca-trust/source/anchors/carc_i101l.crt
update-ca-trust enable
update-ca-trust extract

Verify OCSP Stapling On The Apache HTTPd's Certificate

The first check is making sure Apache's HTTPd is actually stapling to it's TLS certificate the OCSP responses - this can be easily done by using the "openssl s_client" command line utility as follows:

openssl s_client -connect www.lab.carcano.corp:443 -tlsextdebug -status

The Apache's HTTPd answer must contain the stapled OCSP response, as by the first lines in the following snippet (I have cut only the relevant part):

depth=2 C = CH, ST = Tessin, O = Carcano SA, CN = CARC R100L
verify return:1
depth=1 C = CH, O = Carcano SA, CN = CARC I101L
verify return:1
depth=0 C = CH, ST = Tessin, O = Carcano SA, CN = www.lab.carcano.corp
verify return:1
OCSP response:
======================================
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = CH, O = Carcano SA, CN = CARC I101L OCSP
    Produced At: Oct 13 16:50:00 2023 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: 86099856EDC298EB986CA38CBF79DB28B27C9598
      Issuer Key Hash: E33E708DCA69A73D6102E9EF032EB7A0F8A378B4
      Serial Number: 1315B0AFB200C223AEDA04D8E061F42EA84DD00B
    Cert Status: good
    This Update: Oct 13 16:00:00 2023 GMT
    Next Update: Oct 17 16:00:00 2023 GMT
    Signature Algorithm: ecdsa-with-SHA512
    Signature Value:
        30:81:87:02:41:05:d1:53:9c:04:4b:97:ab:21:1a:4b:64:5c:
        cf:45:69:f5:46:96:31:92:77:ed:75:7c:75:79:72:30:e2:17:
        4d:0c:56:96:91:0b:bf:cb:1c:f8:1b:3d:26:3b:03:f8:61:4f:
        4a:78:a2:31:e3:68:d4:7a:38:ef:03:53:03:26:86:e2:02:42:
        01:7a:72:00:6e:1c:c6:ae:28:d6:e4:fb:66:80:46:bd:b6:ed:
        ff:3b:6b:cd:89:63:e1:dc:2a:c9:f0:17:e9:2e:ff:cd:3f:17:
        ed:e5:17:eb:3c:e5:57:6b:0c:34:94:f8:1d:a1:b4:08:12:da:
        3f:16:93:17:eb:1a:55:ec:a2:5f:d9:c2

if instead you get something like the below snippet:

depth=0 C = CH, ST = Tessin, O = Carcano SA, CN = www.lab.carcano.corp
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = CH, ST = Tessin, O = Carcano SA, CN = www.lab.carcano.corp
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 C = CH, ST = Tessin, O = Carcano SA, CN = www.lab.carcano.corp
verify return:1OCSP response: 
======================================
OCSP Response Data:
    OCSP Response Status: trylater (0x3)
======================================

it means the Apache HTTPd server had troubles contacting the CA's OCSP responder and so it was unable to staple a currently valid OCSP response.

This kind of problem can be easily identified also by monitoring log files - when the client connects, Apache tries to contact the CA's OCSP responder: if anything would go wrong,  you'll find entries like the below one in the main SSL error log file ("/var/log/httpd/ssl_error_log"):

[Wed Mar 06 07:01:44.818500 2024] [ssl:error] [pid 10891:tid 10993] (111)Connection refused:
[client 10.211.55.185:42136] AH01974: could not connect to OCSP responder 'ocsp01.lab.carcano.corp:9401'

Apache logs the error also in the Virtual Host's specific error log file ("/var/log/httpd/vhosts/lab.carcano.corp/error.log"):

[Wed Mar 06 07:01:44.818560 2024] [ssl:error] [pid 10891:tid 10993]
AH01941: stapling_renew_response: responder error

Test Access To Public Documents

The next check is to make sure the server correctly provides public documents - or in other words, clients not providing client certificates get the public available data.

To test it, let's first create the index document in the Virtual Host's document root:

cat << EOF > /var/www/vhosts/lab.carcano.corp/www/index.html
<HTML>
<BODY>
<H1>WWW.LAB.CARCANO.CORP</H1>
</BODY>
</HTML>
EOF

Let's now try getting the document root of the "https://www.lab.carcano.corp" Virtual Hosts.

Since it is the index page, it is not necessary to specify it in the URL - just type:

curl https://www.lab.carcano.corp/

The outcome must be as follows:

<HTML>
<BODY>
<H1>WWW.LAB.carcano.corp</H1>
</BODY>
</HTML>

Test Access To Access Restricted Documents

Since we are also providing access restricted contents, we need to make sure restrictions are actually enforced.

The current rule restricts access to the "/restricted" web path only to clients providing a certificate belonging to the "Big Partner Foo SA" (The "O" part of the DN must be "O=Big Partner SA").

To test it, let's first create the "sensitive.txt" document in the Virtual Host's "restricted" directory:

cat << EOF > /var/www/vhosts/lab.carcano.corp/www/restricted/sensitive.txt
This is a sensitive document
EOF

Let's first test an anonymous access to the access restricted document - that means the client does not provide an X.509 client certificate:

curl https://www.lab.carcano.corp/restricted/sensitive.txt

Are you enjoying these high quality free contents on a blog without annoying banners? I like doing this for free, but I also have costs so, if you like these contents and you want to help keeping this website free as it is now, please put your tip in the cup below:

Even a small contribution is always welcome!

The expected outcome is the following error:

curl: (56) OpenSSL SSL_read: error:0A00045C:SSL routines::tlsv13 alert certificate
required, errno 0

So the server correctly answers us that to access the requested URL it is necessary to provide a client certificate to attempt mutual-TLS authentication.

Let's re-try, but this time providing the "~/foo-data-exchange-svc01.crt" certificate and the related "~/foo-data-exchange-svc01.key" private key - the ones of the "Big Partnert SA" "data-exchange-sv01" identity :

curl--cert ~/foo-data-exchange-svc01.crt --key ~/foo-data-exchange-svc01.key \
https://www.lab.carcano.corp/restricted/sensitive.txt

when requested, type the password to unlock the private key and hit enter.

This time it must work, and we must get the document:

This is a sensitive document

The check is not complete yet - it is also mandatory to test access from a client providing a valid certificate (so signed by the "carc i101l" CA, but with a different "O" part in the DN - that means a certificate owned by a different partner/vendor. And of course, last but not least, test access from a client providing an invalid certificate (so NOT signed by the "carc i101l" CA). I leave both these tests as an exercise to the reader.

If instead you get the following error message:

curl: (56) OpenSSL SSL_read: error:0A000410:SSL routines::sslv3
alert handshake failure, errno 0

it can mean that

• the OCSP response for this client certificate has not been generated yet, and so Apache was not able to verify it. Server side it logged the message "AH02263: Re-negotiation handshake failed: Client certificate missing" in the "/var/log/httpd/vhosts/lab.carcano.corp/error.log" log file. In such a scenario you just must wait until the OCSP response for this client certificate gets generated before retrying to connect to the Apache HTTPd server
• the OCSP responder of the CA that signed the client certificate is unreachable. Server side it is logged the message "AH02262: Re-negotiation handshake failed: Client verification failed" in the "/var/log/httpd/vhosts/lab.carcano.corp/error.log" log file and the message "AH01974: could not connect to OCSP responder 'ocsp01.lab.carcano.corp:9401" in the "/var/log/httpd/ssl_error_log" log file.
• it is not the case for our lab, but the above message can be caused also by the client certificate signed by CA not trusted by the Apache HTTPd server's Virtual Host.

Attempting Mutual-TLS With The Revoked Certificate

Since we set up everything to work with OCSP, it is also necessary to test the behavior with revoked certificates.

Let's start by revoking the client's "Big Partnert SA" "data-exchange-sv01"'s certificate using CFSSL - and wait the time to have OCSP responses re-generated.

After revoking it, run a mutual-TLS authenticated connection attempt again to see if the Apache web server denies us access because of the revocation of the certificate.

As we did before, just run:

curl --cert ~/foo-data-exchange-svc01.crt --key ~/foo-data-exchange-svc01.key \
 https://www.lab.carcano.corp/restricted/sensitive.txt

when requested, type the password to unlock the private key and hit enter.

The expected behaviour is Apache denying access and curl showing an error message like that:

curl: (56) OpenSSL SSL_read: error:0A000414:SSL routines::sslv3 alert certificate revoked, errno 0

server side, Apache logs the following error message in the "/var/log/httpd/ssl_error_log" file:

Tue Oct 17 06:44:20.871328 2023] [ssl:error] [pid 3833:tid 3876] [client 127.0.0.1:56190] AH03239: OCSP validation completed, certificate status: revoked (1, 4) [subject: CN=data-exchange-svc01,O=Big Partner Foo SA,C=CH / issuer: CN=CARC I101L,O=Carcano SA,C=CH / serial: 4D1ED13DE091CE3763ED1F9BE99A35A269598BE3 / notbefore: Oct 13 16:08:00 2023 GMT / notafter: Oct 12 16:08:00 2024 GMT]
[Tue Oct 17 06:44:20.871365 2023] [ssl:error] [pid 3833:tid 3876] [client 127.0.0.1:56190] AH02039: Certificate Verification: Error (23): certificate revoked

and the following error message in the "/var/log/httpd/vhosts/lab.carcano.corp/error.log" file:

[Tue Oct 17 07:13:29.340549 2023] [ssl:error] [pid 3835:tid 4029] [client 127.0.0.1:33386] AH02262: Re-negotiation handshake failed: Client verification failed

It also log an access denied (HTTP status 403) in the "/var/log/httpd/vhosts/lab.carcano.corp/access.log" file:

10.4.5.88 - - [17/Oct/2023:06:48:42 +0200] "GET /restricted/sensitive.txt HTTP/1.1" 403 199 "-" "curl/7.76.1"

Footnotes

Here it ends the "Apache HTTPd With Mutual TLS and OCSP Stapling" post, our workshop on strong security on Apache working with mutual-TLS and with OCSP stapling - I hope you understood how all these security mechanisms play together and enable you to configure a very strong set up.

The only requirement to run it is of course running a Cloudflare's PKI and TLS toolkit based PKI: if you want to quickly and easily learn how to set it up and run it, I hope you can enjoy my book "A Full Featured PKI With Cloudflare's PKI and TLS Toolkit" available both as ebook and paperback.

I hate blogs with pop-ups, ads and all the (even worse) other stuff that distracts from the topics you're reading and violates your privacy. I want to offer my readers the best experience possible for free, ... but please be wary that for me it's not really free: on top of the raw costs of running the blog, I usually spend on average 50-60 hours writing each post. I offer all this for free because I think it's nice to help people, but if you think something in this blog has helped you professionally and you want to give concrete support, your contribution is very much appreciated: you can just use the above button.