Kubernetes cert-manager: How to for mTLS,CSI Driver and Helm

Kubernetes cert-manager: How to for mTLS,CSI Driver and Helm post explains how to issue TLS Server certificates and Mutual TLS (mTLS) Client certificates using Kubernetes cert-manager.

We will start with raw YAML manifests to understand the core mechanics of rigid mTLS communication. Then, we will explore how to eliminate long-lived Kubernetes Secrets using the cert-manager CSI Driver, before wrapping up with a practical Helm example to cleanly automate the entire certificate lifecycle for production deployments.

In our previous post, we engineered a fully functional internal PKI simulator using cert-manager and trust-manager. This established a lightweight, automated trust chain perfect for enterprise sandboxes, development environments, and QA testing.

At the end of the post, we completed the groundwork by distributing our trust anchors across the cluster. Because our truststores are now properly configured, the environment is fully equipped to validate certificates issued by our internal CA as well as external third-party certificates whose CAs we explicitly trusted.

With the trust framework sealed, we can now bridge the gap between PKI architecture and daily workload operations.

Issuing Certificates

This section explains how to create a TLS Server Certificate and a TLS Client Certificate suitable to set up mutual TLS authentication.

To familiarize yourself with the registration and certificate issuance process, we will set up a small lab.

Because it is always best practice to operate in a clean and tidy way, we will create a dedicated namespace for our Proof of Concept (PoC) playground:

kubectl create namespace playground

All certificates issued in this lab will belong to this namespace.

Server Certificate

The most straightforward use case is enrolling and issuing a TLS server certificate. This is the most commonly needed certificate type, used to protect end-to-end channels using standard TLS.

Certificates must be described using a YAML manifest. This file provides the necessary identity information to be included in the certificate, its permitted uses, and its lifecycle parameters.

Create the test-server-cert.yml manifest with the following contents:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: foo-tls-srv
  namespace: playground
spec:
  secretName: foo-tls-srv
  isCA: false
  usages:
    - server auth
  dnsNames:
  - "foo.playground.svc.cluster.local"
  duration: 4h 
  renewBefore: 1h
  issuerRef:
    name: issuer.carcano-t1-root01
    kind: ClusterIssuer 
    group: cert-manager.io

it contains the definition of the request for a certificate which:

• is a leaf certificate (isCa: false)
• is explicitly restricted to protecting TLS Servers (usages: server auth)
• is valid for the internal CoreDNS FQDNs of our service (foo.playground.svc.cluster.local). This means it is valid for the foo Kubernetes service in the playground namespace on the Kubernetes's Cluster Network (svc.cluster.local)
• it lasts 4 hours and automatically renews 1 hour before expiration. This short lifespan is chosen intentionally for our lab so you can observe the automatic renewal without waiting days.
• it will be created as foo-tls-srv Certificate CRD in the playground namespace. cert-manager takes also care of automatically creating a copy of it in the foo-tls-srv secret. This secret can then be consumed by pods as a Kubernetes Volume for mounting the certificate and private key in their filesystem
• the certificate will be issued by the issuer.carcano-t1-root01 ClusterIssuer, which uses the carcano-t1 ROOT01 CA's certificate for signing the issued certificates.

In real scenarios, do not set a certificate lifespan that is too short: this is just a lab, so we are intentionally setting a short life to avoid waiting to much time before the automatic renewal. Certificate's lifespan must be set accordingly to regulatory such as the ones from CAB forum.

Once you have created the manifest with the above certificate definition, submit it to Kubernetes by running:

kubectl create -f test-server-cert.yml

the certificate is immediately generated.

You can verify its status by running:

kubectl -n playground get certificate

the output is:

NAME          READY   SECRET        AGE
foo-tls-srv   True    foo-tls-srv   70s

The READY: True status confirms that the issuing process was successful and the keys are bound to the foo-tls-srv Kubernetes secret.

The easiest way to inspect the components of the issued asset (the X.509 certificate, the private key, and the issuing CA certificate) is by fetching them directly from the Kubernetes secret

To extract the public certificate:

kubectl -n playground get secret foo-tls-srv -o jsonpath='{.data.tls\.crt}' | base64 -d

if you want to get the private key:

kubectl -n playground get secret foo-tls-srv -o jsonpath='{.data.tls\.key}' | base64 -d

or, if you want to get the CA's certificate which was used to sign it, run:

kubectl -n playground get secret foo-tls-srv -o jsonpath='{.data.ca\.crt}' | base64 -d

The "ca.crt" file is injected into the secret for absolute convenience; it is a direct copy of the anchor root certificate stored safely within the cert-manager namespace.

To prove that the trust chain is perfectly intact, you can verify the freshly minted certificate against our root CA in real-time using openssl:

openssl verify -CAfile  <(kubectl -n cert-manager get secret carcano-t1-root01-ca -o jsonpath='{.data.ca\.crt}' | base64 -d)  <(kubectl -n playground get secret foo-tls-srv -o jsonpath='{.data.tls\.crt}' | base64 -d)

As expected, the validation returns clean:

/dev/fd/62: OK

Client Certificate

The second use case is generating a Mutual TLS (mTLS) client certificate. This kind of asset is required when the server, during the mTLS handshake, requests the client workload to prove its identity .

Create the test-client-cert.yml manifest file with the following contents:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: bar-tls-client
  namespace: playground
spec:
  secretName: bar-tls-client
  isCA: false
  usages:
    - client auth
  commonName: bar
  subject:
    organizations:
      - Carcano SA
    organizationalUnits:
      - Services
  duration: 4h 
  renewBefore: 1h
  issuerRef:
    name: issuer.carcano-t1-root01
    kind: ClusterIssuer 
    group: cert-manager.io

it contains the definition of the request for a certificate which:

• is a leaf certificate (isCa: false)
• is strictly limited to client-side handshakes (usages: client auth)
• sets the Common Name (CN) to bar and injects explicit X.509 Distinguished Name (DN) attributes via the subject block (organizations and organizationalUnits)
• adheres to the same 4-hour lab lifecycle pattern as our server certificate, allowing quick verification of automated renewals.
• it will be created as bar-tls-client in the playground namespace. cert-manager takes also care of automatically creating a copy of it in the bar-tls-client secret. This secret can then be consumed by pods as a Kubernetes Volume for mounting the certificate and private key in their filesystem
• the certificate will be issued by the issuer.carcano-t1-root01 ClusterIssuer, which as we saw uses the carcano-t1 ROOT01 CA's certificate for signing the issued certificates.

Submit the certificate definition manifest to Kubernetes by running:

kubectl create -f test-client-cert.yml

Mock Up A Mutual-TLS Session

With both the TLS Server and TLS Client certificates successfully issued, we can now simulate a rigid mutual TLS session locally to verify that our trust chain works end-to-end.

First, create the ~/mock temporary directory and change to it:

mkdir ~/mock
cd ~/mock

then, create the test.txt file  dummy payload file:

echo "Hello" > test.txt

To spin up an mTLS-protected endpoint on the fly, we will use openssl s_server.

We will feed it our server credentials directly from the Kubernetes secrets and instruct it to explicitly request and verify a client certificate:

openssl s_server \
  -cert <(kubectl -n playground get secret foo-tls-srv -o jsonpath='{.data.tls\.crt}' | base64 -d) \
  -key <(kubectl -n playground get secret foo-tls-srv -o jsonpath='{.data.tls\.key}' | base64 -d) \
  -CAfile <(kubectl -n playground get secret foo-tls-srv -o jsonpath='{.data.ca\.crt}' | base64 -d) \
  -WWW -port 12345  \
  -verify_return_error -Verify 1

The server process will initialize, dump its parameter details, and block execution on the terminal while listening for incoming client requests:

verify depth is 1, must return a certificate
Using default temp DH parameters
ACCEPT

To complete the mTLS handshake, open a second terminal window.

We will use openssl s_client to connect to our server, passing the client certificate and private key extracted directly from the bar-tls-client Kubernetes secret.

Crucially, we must also pass the root CA certificate via the -CAfile flag so the client can validate the identity of the server.

Run the following command to initiate the connection and request the test.txt payload file:

echo -e 'GET /test.txt HTTP/1.1\r\n\r\n' | \
  openssl s_client \
  -cert <(kubectl -n playground get secret bar-tls-client -o jsonpath='{.data.tls\.crt}' | base64 -d) \
  -key <(kubectl -n playground get secret bar-tls-client -o jsonpath='{.data.tls\.key}' | base64 -d) \
  -CAfile <(kubectl -n playground get secret bar-tls-client -o jsonpath='{.data.ca\.crt}' | base64 -d) \
  -connect localhost:12345 -quiet

If the cryptographic trust chain is fully intact, both terminals will show a successful handshake validation sequence.

Client side, the output should look like as follows:

Can't use SSL_get_servername
depth=1 O = Carcano SA, OU = Services, CN = kube-t1 ROOT01 CA
verify return:1
depth=0 
verify return:1
HTTP/1.0 200 ok
Content-type: text/plain

Hello

As you see, the last line in the output is the contents of the test.txt payload file.

If you attempt to run the same openssl s_client command without passing the client certificate (-cert and -key), or if you present a certificate signed by an untrusted CA, the connection will be immediately terminated with a bad certificate or handshake failure alert.

This confirms that our rigid mTLS validation framework is fully operational.

Avoiding Kubernetes Certificates And Secrets By Using the CSI Driver

Up to this point, we created manifest with the definition of the certificates to be enrolled, submitting them to the Kubernetes API.

This mechanism relies on standard Kubernetes Certificate CRDs, which has a significant security trade-off in production environments: long-lived Kubernetes Secrets.

More specifically, when cert-manager syncs cryptographic assets to a standard Secret, those assets are stored persistently within etcd.

This presents several security challenges:

• Broad RBAC Exposure: any user or service account with broad get or list permissions on Secrets in that namespace can extract highly sensitive private keys.
• Storage Persistence: Private keys remain on disk indefinitely, increasing the attack surface if backup snapshots or etcd nodes are misconfigured.
• Manual Mount Overhead: Pods must explicitly map these secrets as volumes, and tracking the rotation of mounted assets can sometimes introduce subtle synchronization delays.

The cert-manager CSI Driver was specifically designed to mitigate this risk.

This Container Storage Interface (CSI) plugin interacts directly to the cert-manager API: at Pod starts up, instead of reading sensitive data from a persistent Kubernetes Secret, the CSI Driver generates a fresh private key on the worker node and requests a unique certificate on the fly using the cert-manager API. Then, it injects the private key and public certificate directly into a local, in-memory tmpfs volume inside the node.

At Pod destruction time, the private key vanishes from the host entirely — leaving no trace in etcd. To say the least, etcd is involved only at enrollment time to process the CertificateRequest, which is anyway stored only for a few seconds.

The standard safe way to go in security-critical landscapes is to use a KMS connected to the Kubernetes API server that encrypts Kubernetes Secrets at creation time and decrypts them on the fly when requested. When using the cert-manager CSI Driver, cryptographic assets (like private keys) are available only in the memory (tmpfs) of nodes running the specific pods. Even the etcd database contains no sensitive data, as private keys never leave the node and are never stored in Kubernetes Secrets. This means that, if your containerized application supports automatic reload when changes to the certificate and private key are detected, you can use the cert-manager CSI Driver as a valid alternative to an expensive KMS to safely implement Zero-Trust architectures.

In addition to that, the CSI driver also tracks the short-lived asset's (private key and public certificates) expiration and transparently updates the files in the background, making the new version available in the container's filesystem without needing to restart it.

Be aware that Zero Trust can be implemented using the cert-manager CSI Driver alone only if the application running inside the container is able to detect changes in the certificate and private key. Conversely, it is up to you to code the necessary automation to perform a reload (if supported) or to rollout the deployment so that the container is replaced by a new one. Although solutions such as Stakater Reloader exist, they rely on monitoring Kubernetes Secrets, which in turn must be encrypted and decrypted by the Kubernetes API server using an external KMS.

The Certificate's Volume

name: tls-server
 csi:
   driver: csi.cert-manager.io
   readOnly: true

the above snippet illustrates only how to invoke the driver without arguments: a proper invocation must also contain the volumeAttributes dictionary with the information specific for the scenario.

The below snippet illustrates an example for issuing a TLS Server certificate:

volumes:
  - name: tls-server
    csi:
      driver: csi.cert-manager.io
      readOnly: true
      volumeAttributes:
        csi.cert-manager.io/issuer-name: ${CERT_ISSUER_NAME}
        csi.cert-manager.io/issuer-kind: ${CERT_ISSUER_KIND}
        csi.cert-manager.io/common-name: ${COMMON_NAME}
        csi.cert-manager.io/dns-names: ${COMMON_NAME}, ${SVC_NAME}.$(POD_NAMESPACE}.svc.cluster.local
        csi.cert-manager.io/duration: ${CERT_LIFETIME}
        csi.cert-manager.io/renew-before: ${CERT_RENEW_BEFORE}
        csi.cert-manager.io/key-usages: server auth

just replace the ${tokens} with the effective values for your scenario.

• CERT_ISSUER is the name of the issuer to be used to sign the certificate. For example: issuer.carcano-t1-root01
• CERT_ISSUER_KIND is the kind of issuer - for example ClusterIssuer
• COMMON_NAME is the common name you want to assign to the certificate
• SVC_NAME and NAMESPACE are the name of the Kubernetes Service and its namespace - these are used to compose the service FQDN of the cluster service within the Kubernetes Cluster Network, so to enable, TLS inside Kubernetes.
• CERT_LIFETIME is the certificate's duration
• CERT_RENEW_BEFORE is the threshold for the automatic renewal of the certificate.

The Certificate's Volume Mount

Once created a cert-manager's CSI volume, it is of course necessary to provide the mounting information to make the certificate available in the Pod's filesystem, as depicted by the following snippet:

spec:
  template:
    metadata:
    spec:
      containers:
        - name: ${CONTAINER_NAME}
          volumeMounts:
            - mountPath: "/var/run/secrets/tls/server"
              name: tls-server
              readOnly: true

Mastering mTLS and automated lifecycle management is just one small pillar of what is required from a DevOps and DevSecOps Linux professional. If you want to systematically fill your DevOps and DevSecOps knowledge gaps by reading a crash-course like book full of hands-on enterprise exercises, jump directly to the Apress Blueprint Box below to discover how to boost and evolve your career using a self-paced learning path.

Incorporating In An Helm Chart

Managing raw YAML manifests or hardcoded Pod volumes is acceptable for a local sandbox or a quick Proof of Concept. However, in an enterprise production environment driven by GitOps (e.g., ArgoCD or Flux) and Infrastructure as Code (IaC), you need to package and parameterize these configurations. This is where Helm becomes essential.

By creating a reusable Helm chart, you can cleanly abstract your certificate requirements, making your application deployments dynamic, portable, and easily adaptable across multiple environments (Dev, QA, Staging, and Production).

When describing an application consuming certificates with Helm, you have two architectural choices :

• The Traditional Route: Packaging a Certificate CRD alongside your Deployment, allowing cert-manager to provision a standard Kubernetes Secret.
• The Zero-Secret Route: Parameterizing the Deployment's inline CSI volume attributes directly within Helm's templates.

If your security baseline is strict (as it is supposed to be in every productive environment), you must go with the Zero-Secret route.

The following snippet illustrates how to use the Zero-Secret route in a Helm Chart . In this example the Chart's name is  grimoire_kube_bootcamp. 

spec:
  template:
    metadata:
    spec:
      containers:
        - name: foo
          volumeMounts:
            - mountPath: "/var/run/secrets/tls/server"
              name: tls-server
              readOnly: true
            - mountPath: /etc/ssl/certs/clusterca-bundle.pem
              name: clusterca-bundle
              subPath: clusterca-bundle.pem
              readOnly: true
      volumes:
        - name: tls-server
          csi:
            driver: csi.cert-manager.io
            readOnly: true
            volumeAttributes:
              csi.cert-manager.io/issuer-name: {{ .Values.tls.issuer}}
              csi.cert-manager.io/issuer-kind: ClusterIssuer
              csi.cert-manager.io/common-name: '${POD_NAME}'
              csi.cert-manager.io/dns-names: '${POD_NAME}, {{ include "grimoire_kube_bootcamp.fullname" . }}.$(POD_NAMESPACE}.svc.cluster.local'
              csi.cert-manager.io/duration: {{ .Values.tls.duration }}
              csi.cert-manager.io/renew-before: {{ .Values.tls.renewBefore }}
              csi.cert-manager.io/key-usages: server auth
        - name: clusterca-bundle
          configMap:
            name: {{ .Values.tls.trustStoreBundle }}
            defaultMode: 0644
            optional: false
            items:
              - key: clusterca-bundle.pem
                path: clusterca-bundle.pem

The Helm specific part is line 20-30.

More specifically:

• line 21 selects the issuer to be used fro issuing the certificate using the tls.issuer value from the Helm chart's Values file (values.yaml)
• line 23-24: ${POD_NAME} and ${POD_NAMESPACE} are automatically expanded to the name of the Kubernetes' pod name and pod namespace - this substitution is made by the CSI driver itself. Please note the single quotes around '${POD_NAME}.${POD_NAMESPACE}...': this ensures that Helm treats it as a literal string, allowing cert-manager's CSI driver to expand the variables dynamically at runtime without template parsing errors.
• line 24: {{ include "grimoire_kube_bootcamp.fullname" . }} is expanded by Helm to the deployment's name at deploy time
• line 25: set certificate lifetime using the value tls.duration from the Helm chart's Values file
• line 26: set certificate renewal threshold using the value tls.renewBefore from the Helm chart's Values file
• line 30: set the trust-store using the value tls.trustStoreBundle from the Helm chart's Values file

assuming you are in the root directory of the Helm chart directory tree, you can preview the outcome of these substitution at Helm level by running as usual:

helm template .

If it looks as you expect, you can then deploy by running as usual:

helm install .

Footnotes

Here we end our initial exploration of cert-manager: as you see it is a very convenient tool which can really help into setting up and managing a PKI: in addition to provide and easy way for managing certificates lifecycle exploring external CA, it can also be used to set up small CAs using the Simple CA Issuer.

Its deep integration in Kubernetes makes it an invaluable tool for DevSecOps, so professionals cannot miss this skill.

As usual, if you appreciate this post and any other ones, just share this and the others on Linkedin - sharing and comments are an inexpensive way to push me into going on writing - this blog makes sense only if it gets visited.

Also concrete contributions to the maintenance of this blog are also very welcome, ... just put your tip in the below cup.