Professionally administering Linux is a very complex matter: it requires advanced skills on the kernel and on a lot of tools and utilities that are often combined in pipelines and scripts.

Professionals having a thorough understanding of the kernel know how to get optimal performance for specific workloads by tuning the right “knobs” and to what degree to do it. In addition to that they know what commands and tools should be used to gather all the information necessary to fix performance issues. This of course requires having a thorough understanding of the kernel scheduler, of memory allocators, of the virtual memory, the Virtual File System and so.

As anybody would expect, skilled professionals thoroughly know the Linux distribution they use – being able to install and remove software packages is absolutely not enough. They should know:

the file system standard layout used by the distro – so to honour it when installing custom software
how to create custom packages (such as RPM) to ease the installation of custom application and scripts
the peculiarities of the initialization process specific of the distribution, in order to sort out boot issues that may arise sooner or later
how to perform automated installations
how to use filesystems, partitioning and Logical Volume Management utilities to apply changes while keeping online the system

And of course a lot of other stuff.

Apache HTTPd With Mutual TLS and OCSP Stapling

Posted on March 6, 2024August 19, 2024 by Marco Antonio Carcano

TLS peers can verify if a certificate was revoked by checking the CRL (very old and very poorly performing method with lots of shortcomings) or query the OCSP endpoint of the CA that issued the certificate.

However this design still has a shortcoming: what happens if for any reason the OCSP endpoint is unreachable (by accident or by anything caused by the evil people out there)? The outcome is a security risk - if the policy is to deny connection if the OCSP status cannot be checked, you risk to disserve.

Conversely, if the policy is that OCSP status check is a nice to have, there’s a risk that, if a revoked certificate has been stolen by the evils out there, they can just prevent your client to query the OCSP server and hijack the connection to a rogue TLS server managed by them that uses the stolen revoked certificate. To mitigate this you can set up OCSP stapling, which consists of prefetching OCSP responses and attaching them to the X.509 certificate.

In the "Apache HTTPd With Mutual TLS and OCSP Stapling" post we see not only how to configure an Apache server to provide a stapled certificate, but also how to set up mutual TLS authentication, seeing in action what happens when a certificate is revoked.

Cloudflare’s certmgr tutorial – a certmgr howto

Posted on February 26, 2024August 19, 2024 by Marco Antonio Carcano

While running a Public Key Infrastructure (PKI), the maintenance workload due to enrolling new certificates and renewing the existing ones can quickly become overwhelming. Dealing it manually is not only cumbersome: it is frustrating too. Luckily there are ways to automate the enrollment process by providing online Registration Authority endpoints.

Cloudflare's PKI and TLS Toolkit provides both an online Registration Authority as well the client software that can be used to automatically enroll new or renew existing certificates. The aim of the "Cloudflare's certmgr tutorial - a certmgr howto" blog post is ti show how quick and easy is setting up certmgr, the certificate monitoring and automatic enrolling facility provided by Cloudflare.

The operating environment used in this post is Red Hat Enterprise Linux 9 or Rocky Linux 9 - using a different environment may lead to having to adapt or even change things.

Indirect CRL generation, CRL validation and OCSP validation

Posted on June 21, 2023August 19, 2024 by Marco Antonio Carcano

In the "OpenSSL CA tutorial - a full-featured OpenSSL PKI" post we set-up a full featured Public Key Infrastructure with Root and Intermediate Certificate Authorities, Indirect CRL and OCSP Responders. To have a go with that PKI, we also generated an Extended Validation (EV) certificate ("/tmp/foo.crt").

In this post we are using the same PKI we set up in that post, the EV certificate we generated and we also generate a new Organization Validated (OV) certificate: the goal this time is showing how to deal with Indirect CRL generation, CRL validation and OCSP validation.

OpenSSL CA tutorial – A full-featured openssl PKI

Posted on June 1, 2023August 19, 2024 by Marco Antonio Carcano

OpenSSL is a full featured tool capable not only to generate keys and certificates, but also to provide every facility a PKI must have, such as indirect CRL and OCSP responders: these features, along with certificate's best practices such as the Certification Practice Statement (CPS), publishing CRL Distribution Points URL, OCSP Responders URL, CA Issuers URL, are the topics of the OpenSSL CA tutorial - A full-featured openssl PKI.

X509 Certificates Howto – A Public Key Infrastructure Tutorial

Posted on May 6, 2023August 19, 2024 by Marco Antonio Carcano

As we saw in our post on Symmetric And Asymmetric Cryptography, asymmetric key pairs can be used to encrypt and digitally sign documents, but have a huge shortcoming: since they are just keys, they don't provide the metadata necessary to enable people to securely identify their owner. As we saw in the post on GNU Pretty-Good Privacy, GPG addresses this shortcoming by wrapping the key into a container object (the GPG key) that also embeds some metadata about the owner of the key. Of course also the IETF addressed this problem, defining the standard for X.509 certificates (currently X.509v3). X509 Certificates Howto & Public Key Infrastructure Tutorial explores X.509 certificates and how they are trusted by the means of a Public Key Infrastructure.

OpenSSH Tutorial – The Ultimate SSH Guide To Understand It

Posted on February 14, 2023August 19, 2024 by Marco Antonio Carcano

Every system administrator daily use SSH to connect to remote systems and perform they daily tasks: the very most of the time these consist into typing statements on the terminal or copying files from and to the remote system, or again running remote commands, but SSH is much more than this: it not only provides additional facilities such as agent or forwarding, port forwarding and X11 forwarding, but it has also a subsystem that can be exploited to provide SSH secured services such as SFTP.

The goal of the "OpenSSH Tutorial - The Ultimate SSH Guide To Understand It" post is to tell you what historically drove us to SSH, describe the protocol suite in detail and provide a thorough tutorial on using all of these facilities.

SSH is a huge topic: thoroughly explaining both server and client side would require much more than a single post - actually even just explaining server side would deserve several posts. For this reason this post shows only the minimum required settings that are required server side to enable the features that are instead thoroughly described client side. In addition to that, some parts of this post are a little bit redundant, but it was the only way I found to clearly explain how things work from the client perspective and from the server perspective.

This post is based on Red Hat Enterprise Linux 9, but the same concepts apply to the very most of the Linux distributions.