Documenting the project is a critical part of the design process: since there are both business and technical audiences, it is not easy to draw up something that can look appealing and clear to the business and provide both enough implementation details for the technical people.
For this reason, the traditional top-down approach is to write:
- the High Level Design Document: this document is mostly business oriented, describing the project in business terms, highlighting costs, benefits but it must also provide the bare minimum level of technical details to have this business-oriented people be able to figure out the final outcome, as well the necessary details to start writing a more detailed technical document - Personally, I like the approach described in the NASA Systems Engineering Handbook: according to them, writing the High-Level Design Document gets improved and updated while the technical documents that cover and address the low-level requirements get written, since only during that stage a lot of hidden problems are unveiled and addressed: this enable also to provide more accurate estimates by the way.
- the Software Design Document: this is the document actually used to implement the project, so it must be clear enough and provide enough details to enable people working on the project's implementation to work as much independently as possible, avoiding pointless meetings that only lead to to waste their time and delay the project final delivery. Mind this document is quite agnostic and provides only a few hints about deployment - deployments are managed through a different document (each deployment is an instance, so a dedicated document is necessary). If the deployment is small, probably a diagram with a few details and a testing report are enough, but on huge deployments, it is necessary to write a Deployment Plan Document.
Writing both the above documents is a task that is not easy at all, so I wrote this post to provide a template with a use case of a SDD Template Software Design Document For Kubernetes Service - more specifically, a microservice of an existing service.
As you will see, this document leverages and is in compliance with the company wide Software Design Document: this global one is used to provide taxonomy, standards and best practices every project in the corporate must comply with..
As in real-life, let's start from a sample user story submitted by using the Corporate's ticketing system:
As a users I need to publish my XML documents on a server, being able to retrieve and render them on the fly in a human readable format.
Let's see together how to turn this story into a deliverable project, properly analysing and engineering things, drawing up the Software Design Document that can be used throughout the development of the project.
Fancy ERP invoices
Software Design Document
1 - User's Stories
1.1 Initial User Story
As a developer of the "Fancy ERP" project, I need a REST application supporting CRUD operations on XML formatted invoices, providing also search capabilities by "invoice-id", "customer-id" and "issuing date" criteria: if the specified criteria is the document ID, it must be possible to retrieve the document as a pretty printed using a template PDF - all the other kind of searches just return a JSON list with the IDs of the matching invoices. From the security perspective, the document's availability is critical, and the information contained in the document is confidential - sensitive data are the customer name, the customer's VAT-ID, the customer's email and phones/faxes and the customer's address. The required availability is 24x7, with occasional very short services outages tolerated.
2.1 User's Story Analysis
The user's need can be broken down in the following units:
Design Requirements
Solution
REST application supporting CRUD operations
develop the "Fancy XML Invoices" micro-service's CRUD API
be able to retrieve a list of document's ID of documents matching specific search criteria
on push operations, convert the XML document into JSON format, then store the converted version on a Elasticsearch backend - this enable to search for documents matching specific fields
render the XML documents into a human readable format
on retrieve operations, convert the JSON document got from Elasticsearch back to the original XML format and embed on the fly a reference to the related XSLT
turn the invoice it into a PDF document as necessary
on retrieve operations, convert the JSON document obtained from Elasticsearch back to the original XML format and generate on the fly a PDF based on the related XSLT, returning it to the caller.
3 - Project's Specifications
These are the project's specifications that can be inferred by combining the user's story with standards and best practices defined in the "Corporate Wide Software Design Document".
3.1 - Scope
The project's scope is providing a service that can be used to store and retrieve XML formatted invoices.
3.2 - Security Requirements
Since invoices are critical and sensitive data, it is mandatory to protect access to them requiring both authentication and authorization.
The following table summarizes how the Security Requirements claimed in the "Corporate Wide Software Design Document" are addressed within this project.
Corporate's Requirement
Details
services managing sensitive or confidential data must be secured by validating the JWT claim
calls to the REST methods handling the invoices must require and validate a JWT claim signed by the "Fancy ERP login" micro-service. The claim format is described in the main project document "Fancy ERP - Software Design Document".
services managing sensitive or confidential data a must be secured by implementing authorization and event tracking
- calls to the REST methods handling the invoices must make sure the roles provided by the JWT claim grant the access level required to operate on invoices.
- calls to the REST methods handling the invoices must track every CRUD operation and include the user ID in the log entry
log entries must not contain sensitive or confidential data
- log entries must not include values of any of the following fields.
- customer name
- customer's VAT ID
- customer's address
- customer's phones or faxes
- customer's email
role names must be configurable to always fit evolutions the corporate's naming standards
the role's names must not be hard-coded: they must be defined into the settings file so that they can be modified to address future needs
endpoints managing sensitive or confidential data must be TLS protected
the service's endpoint must support and be configured to use TLS
TLS must not be below 1.2
the service's endpoint must support at least TLS v1.2
Production X.509 Certificates must be issued by the Security Team using one of the corporate's production internal CA or one of the allowed public CA
the X.509 certificate and key as well the trust-store used by service must be configurable so to be able to use certificates either issued by public or private Certificate Authorities
3.3 - Availability Requirements
The following table summarizes how the Availability Requirements claimed in the "Corporate Wide Software Design Document" are addressed within this project.
Corporate Requirements
Solution
24x7 services must run on Kubernetes
run the application into a container managed by Kubernetes
microservices containers must have the smallest footprint possible
develop the application either in Golang or C++ and link it statically, so as to be able to run it on a container without other data but its settings and resource files (I.E: container from SCRATCH, leveraging on the OS Kernel only).
avoid developing custom container images only if it is not strictly required, relying on officially supported container images or images without data ("from SCRATCH)"
this is already met by the above solution
implement horizontal scaling as needed
provide guidelines on how to use the collected load metrics for setup autoscaling on Kubernetes
3.4 - Performance Collecting Requirements
The following table summarizes how the Performance Collecting Requirements claimed in the "Corporate Wide Software Design Document" are addressed within this project.
Corporate Requirements
Solution
load metrics must be collected on the corporate's Prometheus server
implement a Prometheus exporter that can be accessed by a Prometheus instance to pull the metrics
3.5 - Monitoring And Tracing Requirements
The following table summarizes how the Monitoring And Tracing Requirements claimed in the "Corporate Wide Software Design Document" are addressed within this project.
Corporate Requirements
Solution
logs messages must be stored in the log aggregator
the REST service must require to supply the "Carcano's HTTP Request Tracking Header", as defined in "Corporate Wide Software Design Document" document and add it to each log entry
calls must be JSON formatted so to be easily trackable using the corporate's log aggregator
use
JSON formatted log files - the format to use is defined in the "Corporate Wide Software Design Document".
3.6 Development Specifications
The following table summarizes the development specifications of this project.
Kind
Requisite
Purpose
Development Language
Golang using Go-Kit and Go-Swagger
Developing The Application
Version Control
Git, using the "Fancy ERP invoices" repository
Managing multiple code versions
Application Runtime Format
Statically linked build
Building
Application Packaging Format
gzipped tarball containing the application along with its defaults configuration files and resource files
Packaging
Application Operational Environment
Container image "from SCRATCH" (Linux Kernel only) , Official Elasticsearch container image for running the document store backend
Running Integration Tests
Container Environment
Containers managed by Podman
Running Integration Tests
3.6.1 - Branching Model
Since this is a microservice of the "Fancy ERP" product, the branching model is the same as the "Fancy ERP" project (Git-Flow).
3.6.2 - Release Versioning
Since this is a microservice of the "Fancy ERP" product, it uses the same versioning standard. Mind that, since it is an independent project, it can manage its own releases, raising versions independently from the main "Fancy ERP'' project.
3.7 - Provisioning's Specifications
This section is used to define the delivery specifications of the project.
3.7.1 - Package's Name
This section is used to define the standard format for package's names:
3.7.1.1 - Testing Packages' Names
In compliance with the claims in the "Corporate Wide Software Design Document", the testing package name must have the following format:
fancy-erp-invoices-<timestamp>-<githash>3.7.1.2 - Release Packages' Names
In compliance with the claims in the "Corporate Wide Software Design Document", the release package name must have the following format:
fancy-erp-invoices-<major>.<minor>.<release_number>3.7.2 - Container's Name
This section is used to define the standard format for the container' names:
3.7.2.1 - Testing Containers' Names
In compliance with the claims in the "Corporate Wide Software Design Document", the testing containers name must have the following format:
fancy-erp-invoices:<timestamp>-<githash>3.7.2.2 - Release Containers' Names
In compliance with the claims in the "Corporate Wide Software Design Document", the release containers' name must have the following format:
fancy-erp-invoices:r-<major>.<minor>.<release_number>3.7.3 - Publishing Specifications
The following table summarizes the publishing specifications.
Kind
Details
Artifacts Repository base URL
https://artifacts-ca-1a.carcano.local
Artifacts publishing path (Integration Tests)
carcano.ch/fancy-erp-invoices/testing
Artifacts publishing path (Releases)
carcano.ch/fancy-erp-invoices/<major>.<minor>
Container Images base URL
containers-ca-1a.carcano.local/carcano
3.7.4 - Deployment Tool
The application must be deployable on Kubernetes using Helm running the "Fancy ERP" 's helm chart – it is so necessary to extend it to deploy the "fancy-erp-invoices" component.
3.8 Runtime Specifications
The following table summarizes the service's runtime environment:
Kind
Details
Application Operational Environment
Dedicated minimal foot-print container image specifically assembled with the application version to run
Container Environment
Containers managed by Kubernetes
Suitable runtimes for running on public clouds
True
4 - LifeCycle
This section describes the overall project’s lifecycle in compliance with the standards and best practices defined in the "Corporate Wide Software Design Document".
4.1 Certificates
The service makes use of certificates - during development and testing certificates must be generated and managed by the developers. Certificates used for running the service in production must be requested to the Security Team according to the conditions defined in the "Certificate Practice Statement" of the issuing Certificate Authority.
The whole production's certificates lifecycle is managed by the Service Owner (or by the Services Manager, if the service has no Service Owner) - this is the only person entitled to request certificates renewals and or revocations: once got the certificate generated by the Security Team, the Service Owner (or the Services Manager) will send the certificate to the DevOps team for having it installed.
4.2 Development
In compliance with the claims in the "Corporate Wide Software Design Document", development is accomplished on the developers' workstations using a dedicated feature branch.
The developer must set up a Makefile with the following targets:
- building the application
- assembling a testing container image that embeds it and its configuration and resource files (the application specifications provides more details about this)
- mocking up a containerised environment providing Elasticsearch and running the unit tests (the lifecycle - development section in the "Corporate Wide Software Design Document" provides the standard to follow for developing this target)
Once the feature's milestone is reached, the developer raises a merge request of the feature branch into the "devel" branch using the process described in the "Corporate Wide Software Design Document".
4.3 Testing
The processes of the Testing stage are described in the "Corporate Wide Software Design Document".
4.4 Publishing A Release
The processes for publishing a release are described in the "Corporate Wide Software Design Document".
4.5 Improvements
Improvements are subjected to raising feature requests that must be challenged and specifically approved by the "Fancy ERP" service owner. Approved features must be added to this document, adding a paragraph beneath "1 - User Stories".
4.6 Decommission
The "Fancy ERP Invoices" will be decommissioned when the main "Fancy ERP" service is decommissioned.
5 - Application's Specifications
This section contains the list of applications that are necessary to develop during this project.
4.1 Fancy ERP Invoices
This is a CRUD API enabling to store and manage XML invoices.
4.1.1 Paths
The service makes use of the following paths
the directory where to lookup for the Configuration File or for the Logging Configuration file - it defaults to the current working directory
4.1.2 Command-line Options
The service accepts the following command line options
the path to the Configuration Directory
4.1.3 JWT Security
The claim succeeds JWT Security authentication if the digital signature can be validated by the specified JWT trust-store.
Authorization levels are granted by checking the roles specified in the "roles" attribute of the claim - grants to each single role are then specified in the YAML configuration file. Valid grants are:
- manage: provides administrative access level
- read: provide read-only access-level to documents retrieved by directly specifying the document-id as search criteria
- search: provide read-only access-level to documents retrieved by specifying any kind of valid search criteria
4.1.4 Logging
Logging must be implemented using the standard logging facility.
The severity of the logged events is as follows:
- INFO - every kind of event useful to track the beginning and end of a request
- WARN - every kind of event caused by an unexpected situation not causing an actual error - for example adding an already existent document or removing a non-existing document
- ERROR - every event causing a request to terminate before completion - for example a connection error to the Elasticsearch backend
- DEBUG - every event that can be used to troubleshoot the execution flow when troubleshooting, so to quickly get close to the block of code where a failure is happening
The runtime severity is set by checking the value of the "LOG_LEVEL" environment variable - if unset, it is automatically set to INFO.
The log formatter must be configured to generate JSON formatted entries - the format to use is defined in the "Corporate Wide Software Design Document".
4.1.5 Configuration File
The configuration file is the "fancy-erp-invoices.yml" file stored in the Configuration Directory. It is a YAML formatted file with the following syntax - the description is provided as comment within the file itself:
port: 8080
xslt_base_url: http://resources-a1.shared-p1.carcano.local/xslts/fancy-erp-invoices-a1
tls:
cert: fancy-erp.crt
key: fancy-erp.key
truststore: ca-trust.crt
jwt:
truststore: jwt-trust.crt
roles:
- role: fancy-erp-operator
grants:
- manage
- role: fancy-erp-user
grants:
- read
- search
elasticsearch:
url: http://elsearch-ca-a1.shared-p1.carcano.local
username: fancy-erp-invoices-a1
password: a.G0odUAn
prometheus:
port: 8090
username: prometheus-ca-a1
password: An0.t3RG00d1
4.1.6 Access Authentication And Authorization
If the JWT claim authentication succeeds, it is then checked that the role provided by the claim matches the name of a role in the Configuration File that grants the proper access level required by the invoking method.
For example, using the sample configuration file (4.1.5), if the required access level is "manage", it is required that the claim provides the "fancy-erp-operator" role.
If anything so far fails, then the Access Authentication And Authorization function outcome is failed.
4.1.7 API (v1.0)
This section describes the"fancy-erp-invoices" RESTful API
4.1.7.1 Service Health
This method is used to get the service health: it must report success only when the service is started. It returns a successful outcome only if the parsing of the settings file had no errors, every specified resource file was loaded without problems, and everything has been properly initialized.
This method does not require neither the Carcano's Requests Tracking HTTP Header nor JWT security.
Example call
GET /api/v1.0/healthzSuccessful Outcome
Returns no data with the HTTP 200 status code.
Failure Outcome
Returns no data with HTTP 500 status code.
4.1.7.2 Service Readiness
This method is used to get the service readiness: it must report success only when all the service dependencies are reachable. The dependency check is triggered every 60 seconds, updating the reported status accordingly. It returns a successful outcome only if every critical dependency is working. It instead returns a failed outcome only if any of the critical dependency is failed.
This method does not require neither the Carcano's Requests Tracking HTTP Header nor JWT security.
Example call
GET /api/v1.0/readyzReturned JSON document
{
"dependencies": [
{
"Elasticsearch": "OK"
}
]
}Successful Outcome
Returns no data with the HTTP 200 status code.
Failure Outcome
Returns no data with HTTP 500 status code.
4.1.7.3 Store Invoice
This method is used to store an invoice XML document.
This method requires both the Carcano's Requests Tracking HTTP Header and JWT security.
Example call
Store the invoice with ID "11223344"
POST /api/v1.0/invoices/11223344Submitted Document
The XML document containing the invoice.
Business Logic
The first step is invoking the Access Authentication And Authorization procedure (4.1.6) specifying that the required access level to run this method is "manage". If the procedure fails, it logs a WARNING event and it immediately returns the Authorization Failed Outcome.
If the Access Authentication And Authorization succeeded, the XML document is validated by checking the linked XSD or DTD: if validation fails or the document does not provide a reference to XSD or DTD, it immediately returns an Invalid Data Outcome.
If the data validation is good, then it converts the document to JSON and pushes it to the Elasticsearch backend: any failure here causes it to immediately return the Failed Outcome.
If the push succeeded, it return the Successful Outcome
Authorization Failed Outcome
Returns no data with the HTTP 403 status code.
Invalid Data Outcome
Returns no data with HTTP 422 status code.
Failure Outcome
Returns no data with HTTP 500 status code.
Successful Outcome
It returns the location header where to download the stored document. For example:
Location /api/v1.0/invoices/11223344Returns the HTTP 200 status code.
4.1.7.3 Retrieve Invoice
This method is used to retrieve a previously stored invoice XML document.
This method requires both the Carcano's Requests Tracking HTTP Header and JWT security.
Example calls
Download the raw invoice with ID "11223344"
GET /api/v1.0/invoices/11223344Download the invoice with ID "11223344", embedding on the fly a reference to the XSLT document that can be used to pretty-print it in a human-readable format
GET /api/v1.0/invoices/11223344?format=with-xsltDownload the invoice with ID "11223344", getting it rendered as a PDF document:
GET /api/v1.0/invoices/11223344?format=pdfBusiness Logic
The first step is invoking the Access Authentication And Authorization procedure (4.1.6) specifying that the required access level to run this method is "read" or "search". If the procedure fails, it logs a WARNING event and it immediately returns the Authorization Failed Outcome.
If the Access Authentication And Authorization succeeded, the Elasticsearch backend is queried to return the document with the specified ID - if it does not exist, then the Not Found Outcome is immediately returned to the caller. Any other error with Elasticsearch causes an entry to be logged as ERROR and the return to the caller the Failure Outcome.
If the document is successfully retrieved, it is converted back to XML. If the URL contains the "format" option, if the value is "with-xslt", a reference to the XSLT is embedded into the converted document. If the value of the "format" option is "pdf", a PDF document is generated on the fly using the XSLT. Any other value of the format field, as well as specifying other fields causes the application to immediately return the Invalid Data Outcome.
Then the application returns a successful outcome to the caller.
Authorization Failed Outcome
Returns no data with the HTTP 403 status code.
Invalid Data Outcome
Returns no data with HTTP 422 status code.
Failure Outcome
Returns no data with HTTP 500 status code.
Successful Outcome
Returns the HTTP 200 status code along with the retrieved document - if it was specified "pdf" as the value of the "format" field, then it is returned the PDF document that was generated on the fly, setting the "application/pdf" as content-type HTTP header.
4.1.7.4 DELETE Invoice
This method is used to delete a previously stored invoice XML document.
This method requires both the Carcano's Requests Tracking HTTP Header and JWT security.
Example calls
Download the invoice with ID "11223344"
DELETE /api/v1.0/invoices/11223344Business Logic
The first step is invoking the Access Authentication And Authorization procedure (4.1.6) specifying that the required access level to run this method is "manage". If the procedure fails, it logs a WARNING event and it immediately returns the Authorization Failed Outcome.
If the Access Authentication And Authorization succeeded, the Elasticsearch backend is queried to delete the document with the specified ID - if it does not exist, then the Not Found Outcome is immediately returned to the caller. Any other error with Elasticsearch causes an entry to be logged as ERROR and the return to the caller the Failure Outcome.
If nothing has failed so far, the application returns a successful outcome to the caller.
Authorization Failed Outcome
Returns no data with the HTTP 403 status code.
Failure Outcome
Returns no data with HTTP 500 status code.
Successful Outcome
Returns no data with the HTTP 200 status code.
4.1.7.4 List Invoices
This method is used to generate a list with the ID of all the stored invoices.
This method requires both the Carcano's Requests Tracking HTTP Header and JWT security.
Example calls
Get the list with the document ID of all the stored invoices:
GET /api/v1.0/invoicesBusiness Logic
The first step is invoking the Access Authentication And Authorization procedure (4.1.6) specifying that the required access level to run this method is "search". If the procedure fails, it logs a WARNING event and it immediately returns the Authorization Failed Outcome.
If the Access Authentication And Authorization succeeded, the Elasticsearch backend is queried to list every document. Any error with Elasticsearch causes an entry to be logged as ERROR and the return to the caller the Failure Outcome.
If nothing has failed so far, the application returns a successful outcome to the caller.
Authorization Failed Outcome
Returns no data with the HTTP 403 status code.
Failure Outcome
Returns no data with HTTP 500 status code.
Successful Outcome
Returns a JSON document with the same format of the following snippet:
{
"invoices": [
"11223344",
"55667788"
]
}The HTTP status code returned is 200.
6 - Deploying On Kubernetes
This section provides the guidelines for deploying the project on Kubernetes.
6.1 - Helm Chart Changes
It is necessary modify the "Fancy ERP" Helm chart adding the missing part for including the "Fancy ERP invoices" in the deployment.
The changes to implement must follow the below specifications.
6.1.1Volume Claims
Add the Volume Claims to bind-mount:
- the "fancy-erp-invoices.yml" config file
6.1.2 - Services
Add the Services to enable horizontal scaling of the following pods:
- fancy-erp-invoices
6.1.3 - Ingresses
Add the Ingress Rules necessary to get proxied to the following pods:
- fancy-erp-invoices - path is "/api/v1/invoices"
6.2 - Deployment Procedure
The procedure to deploy the "Fancy ERP invoices" is the same used to deploy the main project "Fancy ERP":
- add the Corporate's Helm chart repository
- get the "values.yml" file from the chart and configure the settings related to the "Fancy ERP invoices" component
- add the secrets with the credentials for accessing the Elasticsearch service as well as the X.509 certificates and keys and trust-stores
- eventually run the helm install command of the whole "Fancy ERP" project.
Please refer to the "Fancy ERP" Software Design Document for the details.
7 - Testing Procedure
Here we describe the testing procedure to be used after a deployment or a rolling update of the service. Since the "Fancy ERP invoices" service provides both a service health check and a service readiness check endpoints, it is enough to check the outcome of a call to those endpoints. When dealing with a Kubernetes deployment run by using the Helm chart, it is enough to run the Kubernetes statement to get the description of the running pods: both endpoints are indeed monitored by Kubernetes itself, so failures will be highlighted in the output.
7.1 Smoke Tests
Smoke tests can be run using helm with the "test" action - please refer to the "Fancy ERP" Software Design Document for the details. Since smoke tests are actually generating and storing mock invoices, the invoice holder MUST be any of the mock customers defined in the "Fancy ERP" Software Design Document, and the services and goods must be the mock services and goods defined in the "Fancy ERP" Software Design Document. At the end of the smoke tests, these mock invoices must be automatically deleted.
Footnotes
Here it ends this post dedicated to writing a clean and easy to understand Software Design Document. I hope you found it useful.
