Cloudflare's certmgr tutorial

While running a Public Key Infrastructure (PKI), the maintenance workload due to enrolling new certificates and renewing the existing ones can quickly become overwhelming. Dealing it manually is not only cumbersome: it is frustrating too. Luckily there are ways to automate the enrollment process by providing online Registration Authority endpoints.

Cloudflare's PKI and TLS Toolkit provides both an online Registration Authority as well the client software that can be used to automatically enroll new or renew existing certificates. The aim of the "Cloudflare's certmgr tutorial - a certmgr howto" blog post is ti show how quick and easy is setting up certmgr, the certificate monitoring and automatic enrolling facility provided by Cloudflare.

The operating environment used in this post is Red Hat Enterprise Linux 9 or Rocky Linux 9 - using a different environment may lead to having to adapt or even change things.

This post is a sample from the book "A Full Featured PKI With Cloudflare's PKI and TLS Toolkit": if you are interested into a quick deep dive on this topic and be able to quickly set up a full featured nearly real-life lab with it, here is the link to the ebook and here the link to the paperback.

The Context

In this post we see how to install and configure the certmgr to track the certificate ("/etc/cfssl/tier01/ra/certs/crt.pem"), automatically renewing it along with its private key ("/etc/cfssl/tier01/ra/keys/key.pem") right before its expiration. Just to provide some context, in the lab described in the "A Full Featured PKI With Cloudflare's PKI and TLS Toolkit" that is the certificate of the online Registration Authority endpoint itself.

You can of course easily re-adapt what explained here for example to have certmgr tracking any TLS server's certificate, such as the one of a web server, of a mail server and so on. The only things you must be clear about is that certmgr must be installed and running on the host where the TLS server you want to automatically manage the certificate's lifecycle is running.

Overview

Certmgr loads its settings from JSON formatted manifests describing the needed certificates: this means that certmgr can manage the lifecycle of multiple certificates at the same time.

For each certificate described in the manifests, certmgr:

if the certificate does not exist yet, it generates first the private key, then the JSON formatted CSR and submits it to the CFSSL multi-CA Registration Authority endpoint
if the certificate exists, and the renewal threshold is hit, it automatically enrolls a new one, reloading the related TLS service (if set up to do so)

Mind that certmgr monitors the certificate existence calling the CFSSL's online Registration Authority, ... this means for example that it won't renew a non-expiring certificate only because you deleted the file from the filesystem by mistake. You must be wary also that it (correctly) does not automatically enroll a new certificate if the current one has been revoked.

Installing

Only during the installation and setup, we work as the "root" user, since almost every statement require administrative privileges and we are aware of what we are doing, it is just pointless having to prepend "sudo" at each one of them:

sudo su -

In compliance with the principle of running services with the least amount possible of privileges, we create the "cloudflare" service user and use it to run the services:

adduser -r -m -c "CloudFlare's PKI Toolkit Service User" -s /sbin/nologin cloudflare

Let's set a convenient umask so to have permission properly set when creating files and directories – this spare us from continuously having to type "chmod" statements:

umask 022

We are now ready for install certmgr.

First, decide on the certmgr’s version to install - in this post it is "3.0.3" – let’s set it into the "CERTMGR_VERSION" variable as follows:

CERTMGR_VERSION=3.0.3

The package must be owned by the "root" user, granting other users only the execution right – this way even the "cloudflare" user is not able to alter neither the binaries nor the settings.

Let's start by creating the installation directory tree beneath the "/opt/certmgr" directory:

CERTMGR_ROOT=/opt/certmgr
mkdir -m 755 ${CERTMGR_ROOT} \
${CERTMGR_ROOT}/${CERTMGR_VERSION} \
${CERTMGR_ROOT}/${CERTMGR_VERSION}/sbin \
${CERTMGR_ROOT}/${CERTMGR_VERSION}/etc

To enable keeping multiple certmgr’s versions, we also create the "current" symlink to the directory tree with the version we want to run:

ln -s ${CERTMGR_ROOT}/${CERTMGR_VERSION} ${CERTMGR_ROOT}/current

Lastly let's symlink the "etc" directory to "/etc/certmgr":

ln -s ${CERTMGR_ROOT}/${CERTMGR_VERSION}/etc /etc/certmgr

set it up to be available system-wide:

ln -s ${CERTMGR_ROOT}/current/sbin/certmgr /sbin/certmgr

Installing The Pre-built binaries

This is the easiest and quickest way of installing the Cloudflare’s certmgr.

First, set the machines’ architecture in the "ARCH" variable as follows:

ARCH=amd64

If you are working on an ARM, set ARCH=arm64.

Download the certmgr command line utility as follows:

ARCH=amd64
DOWNLOAD_URL=https://github.com/cloudflare/certmgr/releases/download
curl -L ${DOWNLOAD_URL}/v${CERTMGR_VERSION}/certmgr-linux-${ARCH}-v${CERTMGR_VERSION}.tar.gz \
| tar -C ${CERTMGR_ROOT}/${CERTMGR_VERSION}/sbin -xz
mv ${CERTMGR_ROOT}/${CERTMGR_VERSION}/sbin/certmgr-linux-${ARCH}-v${CERTMGR_VERSION} \
${CERTMGR_ROOT}/${CERTMGR_VERSION}/sbin/certmgr
chmod 755 ${CERTMGR_ROOT}/${CERTMGR_VERSION}/sbin/certmgr

Building From Sources

You can of course build the Cloudflare’s certmgr from sources: this way is useful when there is no pre-built binary for the architecture you are using, or if you are customizing it.

Since the PKI Toolkit is written in Golang, you need to download it (mind here we need to work as the “root” user) – first decide on the Golang version you want to use, for example:

GO_VERSION=20.3

then download and install Golang using the following statement:

 wget -qO- https://go.dev/dl/go1.${GO_VERSION}.linux-arm64.tar.gz | tar xvz -C /usr/local

since it is also needed git, install it as follows:

dnf install -y git

Since the best practice is not to use the root user when building and packaging software, now we must temporarily switch to the "cloudflare" user:

sudo -u cloudflare bash

Before going on, add "/usr/local/go/bin" to the PATH environment variable in the "~/.bashrc" file, then source it or disconnect the session and become the "cloudflare" user again.

We are ready to download and build certmgr - just type:

cd ~
go install github.com/cloudflare/certmgr/cmd/...@latest

guess the version we have just built:

~/go/bin/certmgr version

in this example I got version is 1.6.5 (that is a very old one by the way), ... for the sake of completeness, let's see how to install it system:

CERTMGR_VERSION=1.6.5
mkdir -p ${CERTMGR_ROOT}/${CERTMGR_VERSION}/sbin
# uncomment only if you really want to use this old version
#[ -L /opt/certmgr/current ] && rm -f /opt/certmgr/current
#ln -s ${CERTMGR_ROOT}/${CERTMGR_VERSION} ${CERTMGR_ROOT}/current
cp /home/cloudflare/go/bin/certmgr ${CERTMGR_ROOT}/${CERTMGR_VERSION}/sbin

Basic Settings

Configuring certmgr is quick and easy - create the certmgr’s configuration directory tree:

 mkdir -m 755 /etc/certmgr/conf.d

then setup the certmgr’s defaults file "/etc/certmgr/certmgr.yml":

dir: /etc/certmgr/conf.d
default_remote: https://ca01.lab.carcano.corp:9800
svcmgr: systemd
before: 72h
interval: 720m
metrics_port: 8080
metrics_address: 10.21.34.52
take_actions_only_if_running: true

this file provides reasonable defaults to certmgr so to avoid explicitly setting them in each certificate configuration file.
The above file sets the "/etc/certmgr/conf.d" as the directory where to look for certificate configuration files and assigns the following defaults.

CFSSL Multi CA Registration Authority endpoint: "https://ca01.lab.carcano.corp:9800"
Process that manages the monitored services: systemd
Certificate renewal threshold: 72 hours before expiration
Checking interval: every 720 minutes (12 hours)
Port for exporting Prometheus metrics that can be collected by: 8080
Interface to bind the metrics endpoint to: 10.21.34.52 - the IP address of the server we installed certmgr on to
Take actions (such as renewals and restart of the service) on a monitored certificate only if the
monitored service is running.

It is then necessary to set up the certmgr Systemd unit – create the "/etc/systemd/system/certmgr.service" file with the following contents:

[Unit]
Description=Cloudflares' CertMgr

[Service]
Type=simple
Restart=on-failure
RestartSec=5
User=root
Group=root
ExecStart=/opt/certmgr/current/sbin/certmgr -f /etc/certmgr/certmgr.yml

[Install]
WantedBy=multi-user.target

And reload Systemd as usual:

systemctl daemon-reload

Monitoring A Certificate

We are finally ready to have the first certificate monitored by certmgr. We just need to add the manifest file describing the certificate to monitor beneath the "/etc/certmgr/conf.d" directory : configure the "/etc/certmgr/conf.d/ca01.lab.carcano.corp.json" file as follows:

cat << EOF > /etc/certmgr/conf.d/ca01.lab.carcano.corp.json
{
    "service": "cfssl-mra@tier01",
    "action": "restart",
    "request": {
        "CN": "ca01.lab.carcano.corp",
        "key": {
            "algo": "ecdsa",
            "size": 256
        },
        "hosts": [
            "ca01.lab.carcano.corp",
            "10.21.34.52"
        ],
"names": [ {
                "C": "CH",
                "ST": "Tessin",
                "O": "Carcano SA"
} ]
    },
    "private_key": {
        "path": "/etc/cfssl/tier01/ra/keys/key.pem",
        "owner": "root",
        "group": "cloudflare",
        "mode": "0640"
    },
    "certificate": {
        "path": "/etc/cfssl/tier01/ra/certs/crt.pem",
        "owner": "root",
        "group": "root",
        "mode": "0644",
        "key_usages": [
            "signing",
            "key encipherment",
            "server auth"
        ]
    },
    "authority": {
        "auth_key": "${AUTH_KEY}",
        "label": "carc_r100l",
        "profile": "server",
        "root_ca": "/etc/cfssl/tier01/carc_r100l/certs/ca.crt"
} }
EOF

The above certificate and key manifest require certmgr to perform the following tasks:

Make sure the following files exist:
- the ecdsa256 "/etc/cfssl/tier01/ra/keys/key.pem" key file, owned by the "root" user and "cloudflare" group, with Posix permission 0640
- the "/etc/cfssl/tier01/ra/certs/crt.pem" x509 certificate file, owned by the "root" user and "root" group, with Posix permission 0644
Make sure the certificate matches the following criteria:
- CN is "ca01.lab.carcano.corp"
- Subject String is "C=CH , ST=Tessin, O=Carcano SA"
- Subject Alternative Names are:
  - ca01.lab.carcano.corp
  - 10.21.34.52
- key usages are "signing", "key encipherment" and "server auth"
Make sure the certificate is not expired, using the value of the "before" attribute of the file "/etc/certmgr/certmgr.yml" as the expiration threshold.

If the above conditions are not met, certmgr generates a key and submits the Certificate Signing Request to the multi-Certificate Authority endpoint set as "default_remote" in the "/etc/certmgr/certmgr.yml" file ("https://ca01.lab.carcano.corp:9800"), selecting the "carc_r100l" CA, using the "server" profile.

The TLS certificate of the multi-Certificate Authority endpoint is validated using the "/etc/cfssl/tier01/carc_r100l/certs/ca.crt" trust-store file.

Now that we have at least one monitored certificate we can start the certmgr service and configure it to load at system boot:

systemctl enable --now certmgr

If everything is properly set, you must find entries such as the following ones in the "/var/log/messages" file:

Aug 2 07:50:05 localhost certmgr[5383]: 2023/08/02 07:50:05 [INFO] manager: processing certificate spec /CARC R100L CRL/C=CH/O=Carcano SA/ST=Tessin (attempt 1)
Aug 2 07:50:05 localhost certmgr[5383]: 2023/08/02 07:50:05 [INFO] encoded CSR
Aug 2 07:50:05 localhost certmgr[5383]: 2023/08/02 07:50:05 [INFO] manager: executing configured action due to change type key for /etc/cfssl/tier01/ra/certs/crt.pem
Aug  2 07:50:05 localhost certmgr[5383]: 2023/08/02 07:50:05 [INFO] restarting service cfssl-ra@carc_r100l
Aug  2 07:50:05 localhost certmgr[5383]: 2023/08/02 07:50:05 [INFO] manager: certificate successfully processed

And of course, the "/etc/cfssl/tier01/ra/keys/key.pem" and "/etc/cfssl/tier01/ra/certs/crt.pem" files are automatically created.

Please note how the service “cfssl-ra@carc_r100l” related to the monitored certificate was automatically restarted.

The entire set of available settings can be found on the README.md document in the certmgr git repo on GitHub - https://github.com/cloudflare/certmgr.

Footnotes

Here it ends this post dedicated to Cloudflare's certmgr - as you see it is a really handy and easy to set up tool. The only requirement to run it is of course running a Cloudflare's PKI and TLS toolkit based PKI: if you want to quickly and easily learn how to set it up and run it, I hope you can enjoy my book "A Full Featured PKI With Cloudflare's PKI and TLS Toolkit" available both as ebook and paperback.

Writing a post like this takes a lot of hours. I'm doing it for the only pleasure of sharing knowledge and thoughts, but all of this does not come for free: it is a time consuming volunteering task. This blog is not affiliated to anybody, does not show advertisements nor sells data of visitors. The only goal of this blog is to make ideas flow. So please, if you liked this post, spend a little of your time to share it on Linkedin or Twitter using the buttons below: seeing that posts are actually read is the only way I have to understand if I'm really sharing thoughts or if I'm just wasting time and I'd better give up.

Cloudflare’s certmgr tutorial – a certmgr howto