TLS (Formerly SSL) | Wikipedia | RFC-6066 | TLS 1.3 | TLS 1.2

Overview

TLS/SSL, "Transport Layer Security" FKA "Secure Socket Layer", is a trust-based PKI (Public Key Infrastructure) scheme of HTTPS protocol, extending HTTP to secure the internet. The server sends a single cryptographically-generated certificate if signed by a root CA, else a chain of them. The client application validates each server certificate against the Certificate Authority (CA) that supposedly signed it.

Nominally, the client application has access to a trust store containing a list of trustworthy-CA certificates, which it uses to verify the (supposedly) CA-signed server certificate or chain of certificates. CAs are hierarchical, with intermediate-CA certificates forming a chain of trust from end-entity (server) certificate to the root (CA) certificate.

A certificate's Issuer is the CA that signed that Subject certificate, so the root (CA) is always self-signed. However, the term "self-signed" more often references an end-entity certificate created sans CA. These are somewhat useful, but only for test/dev purposes. Care must be taken even within that scope because browsers and other clients respond entirely differently when the end-entity (server) certificate sent during the TLS handshake does not validate against any client-trusted CA.

POSIX-type operating systems have a directory of such certificate files. Linux stores them in the /etc/ssl/certs directory. The ca-certificates.crt file contains a concatenated list of such certificates. Other CA root-certificate files (*.crt) may exist in that directory as standalones, e.g., root-ca-site.local.crt .

Windows has an entirely different scheme of course; Certificates.msc

TLS Handshake

HTTPS is an extention of HTTP. An HTTPS connection starts with a TLS client/server handshake whereof the certificate is sent/validated and the TLS parameters (cipher suite and such) are set (negotiated if allowed). Otherwise, the TLS-handshake fails and the connection is terminated.

On TLS-handshake failure, the server is likely to send a "handshake failure" alert. Such may occur if the certificate's CA cannot be validated by the client, or if client and server have no mutually supported TLS-ciphers suite. The TLS handshake failure is not an HTTP-level response; it's part of the TLS protocol. The HTTP layer comes into play only after a secure TLS connection has been established. If the TLS handshake fails, the HTTP layer doesn't have an opportunity to send an HTTP response code because the connection hasn't been established.

OpenSSL

The industry-standard library and its CLI utility for all things TLS.

The OpenSSL utility, openssl, is a multi-purpose tool used to create, parse, validate, convert formats, and test TLS keys, certificates, and such.

References:

Glossary

Transport Layer Security (TLS)

Transport Layer Security is the newer security protocol that replaced Secure Sockets Layer (SSL). Though the SSL protocol is obsolete and has been so for about a quarter century, the term "SSL" stubbornly remains with us, e.g., "OpenSSL". Engineers often refer to TLS as SSL, while actual SSL (protocols, ciphers, certificates and keys) are extremely insecure and entirely inadequate. Never use actual SSL.

Encodings v. Formats

Reference:

PEM (Privacy Enhanced Mail)

Base64 encoded (ASCII) file format. PEM is the most common "format" (encoding) for TLS certificate and key files; widely used in web servers and other network applications where textual configuration files are standard. Common extensions for PEM files are: .crt, .cer, and .pem if the document is a public certificate; .key if it's a (private) key. Do not rely on the filename extension to indicate its actual format.

Many cryptography standards use ASN.1 to define their data structures, and DER to serialize those structures. Because DER produces binary output, it can be challenging to transmit the resulting files through systems, like electronic mail, that only support ASCII.

PEM certificates include a header and footer:

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

A certificate file may contain many such concatenated certificates. The most common certificate files are those of either a list of root CA certificates AKA trust-store AKA ca-bundle file, or a certificate chain AKA full-chain certificate. The latter is commonly referred to as the "server certificate".

The full-chain (of trust) certificates must be properly ordered, whereof the first cerificate is that of the end-entity (server). Each cerificate thereafter is that of the prior's Issuer; the CA that signed (and therefore validates) the preceding certificate. This chain goes all the way back to, but not (typically) including, the root-CA certificate.

That root-CA certificate is expected to be in the clients' trust store. The root-CA certificate is distinguished by having identical Subject and Issuer, AKA it is a Self-signed Certificate of a trustworthy Certificate Authority.

Many command-line tools, including OpenSSL's (openssl), read PEM files. It is especially common in Unix/Linux environments.

DER (Distinguished Encoding Rules)

A binary encoding scheme; a subset of BER (X.690) that eliminates some of the superset's flexibility to guarantee there is one and only one way to encode a message. That is, to encode, decode, re-encode, re-decode, and re-re-encode a DER message, all those encodings remain identical.

DER is commonly used in security-related applications such as X.509 digital certificates (and keys). Common extensions are: .crt, .cer, and .der if a public certificate; .key if a private key, so do not rely on the file's extension to indicate format. Not many applications require DER formatted certificates or keys, but this binary form is useful for conversions between otherwise incompmatible ASCII-based encodings.

# Convert format: DER to PEM (From binary to human-readable text) 
openssl x509 -inform der -in certificate.der -outform pem -out certificate.pem

# Convert format: PEM to DER (From human-readable text to binary) 
openssl x509 -in certificate.pem -outform der -out certificate.der

JKS (Java KeyStore)

A proprietary file format developed by Oracle for storing cryptographic keys and certificates securely. It is primarily used in Java applications to manage and secure keys and certificates for TLS and other uses (Document Signing, Software Integrity Checking, Internal Communications, …).

Also, take care to not mistake JKS with similar acronyms regarding the unrelated JSON-formatted PKI, such as JWKS, JWK, ….

PKCS (Public-Key Cryptography Standards)

PKCS encompasses a group of standards, including specifications for various formats and protocols designed for secure cryptographic communication; encrypted data, private keys, and public key certificates, among other things.

PKCS#1

Defines the format for RSA encryption and signing. It specifies how to encode data for signing or encryption with RSA.

PKCS#7/P7B/CMS

A standard (RFC-5652) for encryption and signing, suitable for messages like signed or encrypted emails. It defines a general syntax for data that may have cryptography applied to it, such as digital signatures and encryption. It is widely used in various security and cryptographic applications. This standard is versatile, supporting not only encryption and digital signatures but also other enhancements like data compression and certificate dissemination.

One or more Base64 ASCII certificates are stored in a .p7b or .p7c file. Commonly used in Windows and Java applications.

Windows Server's Certificate Services (AD CS) returns the fullchain certificate (per approved CSR) in P7B format.

## Convert from P7B to PEM
openssl pkcs7 -print_certs -in input.p7b -out output.pem
PKCS#8

Specifies the private key information syntax, which is a format for storing private keys; both encrypted and unencrypted forms.

PKCS#10/CSR

A format regarding Certificate Signing Request (CSR) data structure, for requesting certificates from a CA. It specifies what information the CSR contains and how that information is organized. This (ASN.1) format is recognized universally and is the standard format for CSRs regardless of the software or platform being used. The file may be in either DER or PEM format. Common filename extensions for a CSR are .csr, .pem and .der.

PKCS#12/PFX/P12

A binary archive-file format for storing many cryptographic objects (certificates and key) in a single password-protected file, each in a SafeBag (container); successor to Microsoft's PFX. Common filename extensions are .pfx and .p12. Often used in Java applications and across Windows machines. Code-signing certificates stored in PFX are also often used for signing with Microsoft Authenticode.

Key characteristics and uses of PKCS#12 include:

CMC (Certificate Management over CMS)

A standard (RFC-5272/RFC-5273) for certificate management that allows for various operations, including requesting certificate issuance, renewal, and revocation.

A CMC-formatted CSR is essentially a more complex and versatile request format compared to the traditional PKCS#10 CSR. It is designed to support a wider range of certificate management functions and to facilitate more sophisticated interactions between entities requesting certificates (clients) and the entities issuing them (CAs).

CMC-formatted CSRs are particularly useful in environments requiring advanced certificate management capabilities, especially where multiple certificate-related operations need to be performed in a streamlined and secure manner. Due to its complexity and advanced features, CMC is often used in enterprise environments, in systems with large-scale certificate management needs, or in scenarios requiring automated certificate lifecycle management.

CMC @ RHEL

Key

AKA Private Key; the private artifact of an asymmetric (public-private) key pair creation. The server's key is private; a secret that must be well protected, never shared, and sufficiently rotated. The public artifact is the server's certificate.

Certificate

AKA Public Key Certificate, which is composed of the public artifact (AKA Public Key) of an asymmetric (public-private) key pair creation, plus metadata extracted from the CSR. The certificate is public; presented by a web server whenever a new TLS connection is requested. The certificate contains its Subject host name (FQDN), which must match the FQDN of server sending it, and the Issuer.

The certificate's Issuer is the CA that signed it. Every CA has a certificate, which is used to verify the server's CA-signed certificate. If the Issuer is not a Root CA, then it must be an Intermediate CA, and so the chain of trust may continue until the Root-CA certificate is obtained and used to verify the signature of the preceding Intermediate-CA certificate in the chain of certificates sent to the client. All such cert bundled into one file and typically referred to simply as the "server certificate".

SAN (Subject Alternative Name)

Extenstion for CN (Common Name) AKA Domain Name; a field of a certificate that indicates for which domain(s) the certificate is valid. A single certificate may contain many SANs. That is, may be valid for many different domain names.

Wildcard Certificate

The asterisk character (*) is the wildcard, and can be substituted with any valid hostname. Instead of being issued for a specific Fully Qualified Domain Name (FQDN), such as app.example.com, wildcard certs are valid for a whole range of subdomain names. So a cert issued to *.example.com would cover app.example.com and cdn.example.com, yet not other.cdn.example.com.

Certificate Authority (CA)

A certificate authority verifies details about a domain owner’s request for certificates. Only after validating the certificate-signing request (CSR) do CAs sign and issue the server certificate; typically returning the full-chain (of trust) certificate. Browsers and operating systems maintain a list of trusted certificate authorities. If a server certificate is signed by one of these trusted CAs, it will also be trusted. There are several organizations and forums that maintain lists of trustworthy CAs; it is fragmented.

Full-Chain Certificate

AKA "server certificate" AKA "certificate chain" file.

A "full chain" certificate refers to a certificate file containing the entire chain of trust; the end-entity (server) certificate and all the intermediate certificate(s) up to, but not icluding, the root CA certificate. This chain of certificates helps establish trust from the end-entity certificate back to a trusted root certificate authority (CA). Most CAs default to this "bundle" or "full chain" option when delivering certificates.

Order matters. The first certificate listed is that of the end-entity (server). The second certificate (intermidiary) is that of the CA that signed (and therefore can verify) the 1st, the 3rd verifies the 2nd, and so on. So, the last (Nth) intermidate certificate would be validated by the root CA, which must reside in the clients' (TLS) trust store.

Certificate Chain Trust: The server is responsible for sending its own certificate and any intermediate certificates necessary to form a complete path to a root certificate that the client trusts. The inclusion of intermediate certificates is crucial because clients might not have them, unlike root certificates, which are expected to be pre-installed in the client's certificate store.

# Get the (full-chain) certificate of a server ($h) at its port ($p)
h='google.com' # Host
p='443'        # Port
openssl s_client -connect $h:$p -showcerts < /dev/null > ${h}_${p}.full-chain.crt

# Verify the server's ca-signed certificate ($any.crt) against the CA ($ca.crt) that signed it
# (The CA file may be a trust-store bundle; a concatenated list of CA certs in PEM format.)
openssl verify -CAfile $ca.crt $any.crt

Certificate hierarchy:

Intermediate Certificate (example)

CN: GTS CA 1C3

...                                         
 1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3          
   i:C = US, O = Google Trust Services LLC, CN = GTS Root R1         
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256             
   v:NotBefore: Aug 13 00:00:42 2020 GMT; NotAfter: Sep 30 00:00:42 2
-----BEGIN CERTIFICATE-----                                          
MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQsw     
...     
1IXNDw9bg1kWRxYtnCQ6yICmJhSFm/Y3m6xv+cXDBlHz4n/FsRC6UfTd             
-----END CERTIFICATE-----

When a client, such as a web browser, connects to a server secured with SSL/TLS, it checks the certificate chain to ensure that the end-entity (server) certificate is valid and signed by a trusted intermediate certificate, which, in turn, is signed by a trusted root certificate. That is, the client must have its own set of trusted-CA certificates, or use that of the system (OS) in which it is running, against which to validate the server's certificate.

Including the full chain when configuring SSL/TLS on a server is important to ensure that clients can validate the server's certificate properly. Without the full chain, clients may not be able to establish the chain of trust, and the connection could be deemed untrusted.

X.509

A standard that defines the format of public key certificates. These certificates are used in various security and networking protocols, including TLS/SSL, which is the basis for secure connections on the internet. The X.509 standard is part of the X.500 series of standards defined by the ITU-T (International Telecommunication Union Telecommunication Standardization Sector) for the Directory Services.

Key features of X.509 certificates include:

X.509 certificates play a crucial role in establishing trust in digital communications. By verifying the chain of trust from a given certificate up to a trusted root CA, entities can establish secure communication channels. This trust model underpins the security of HTTPS websites, email encryption (S/MIME), and many other protocols that secure data exchange over networks.

CRL (Certificate Revocation List)

Certificates may include information on how to access a certificate revocation list. Clients will download and check this list to make sure the certificate has not been revoked. The CRL is a list or database of TLS certificates (serial numbers) that have been revoked by the issuing CA before their scheduled expiration date and should no longer be trusted. CRLs are slowly being depricated as alternate certificate revocation technologies (such as OCSP responders) are increasingly used instead. Nevertheless, CRLs are still widely used by the CAs.

OCSP (Online Certificate Status Protocol)

The OCSP protocol is a TLS extension; a replacement for CRLs, with the benefits of being more real-time and requiring less bandwidth. The general operation is similar: clients are to query the OCSP responder to check if a certificate has been revoked. OCSP is an internet protocol (RFC-6960) allowing applications to check the revocation status of a certificate in real-time without needing to download and parse a CRL. However, OCSP also introduces its own complexities, such as the need for additional infrastructure to handle real-time requests.

OSCP Stapling

The server queries the OCSP responder at regular intervals and then "staples" the response to the handshake process when a client connects. This method reduces the performance impact, alleviates privacy concerns, and ensures that clients receive up-to-date revocation information without requiring separate OCSP queries.

Formally known as the TLS Certificate Status Request extension (RFC-6066), OSCP Stapling is a standard for checking the revocation status of X.509 digital certificates. It allows the presenter of a certificate to bear the resource cost involved in providing Online Certificate Status Protocol (OCSP) responses by appending ("stapling") a time-stamped OCSP response signed by the CA to the initial TLS handshake, eliminating the need for clients to contact the CA, with the aim of improving both security and performance.

Domain Validation (DV)

A domain validated certificate will be issued to somebody who has proven they control the domain name requested for the certificate. This proof often takes the form of serving a unique token from your web server or DNS records, which the CA will check for before issuing the certificate.

Organization Validation (OV)

An organization validated certificate means that the certificate authority also verified the company name and address in public databases. This information is put into the certificate, and is typically displayed only when the user clicks the green padlock icon to investigate further.

Extended Validation (EV)

Extended validation is more thorough than domain or organization validation. EV certificates are issued after checking not only domain ownership, but also verifying the existence and location of the legal entity requesting the certificate, and that said entity controls the domain being verified.

Unlike DV and OV certificates, EV cannot be issued as a wildcard certificate.

EV certificate also gets special treatment in web browsers. Whereas browsers typically denote a DV certificate with a green-padlock icon, EV certificates also show a larger green bar containing the name of the organization it was issued to. This is intended to reduce phishing attacks, though some studies show that users tend not to notice when this green bar is missing.

Intermediate CA

When creating an Intermediate Certificate Authority (CA) to scope its authority to a specific subset of CIDRs, IPs, hostnames, or other identifiers, you need to carefully configure the certificate's extensions and constraints. Here are the key parameters and considerations:

How to configure an Intermediate CA using OpenSSL:

  1. OpenSSL Configuration File (openssl.cnf):

    [ v3_intermediate_ca ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:TRUE, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign
nameConstraints = critical, permitted;DNS:example.com, permitted;IP:192.168.1.0/24

  1. Generate the Intermediate CA Certificate:

    openssl req -new -key intermediate.key -out intermediate.csr
openssl x509 -req -in intermediate.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out intermediate.crt -days 365 -extfile openssl.cnf -extensions v3_intermediate_ca

Projects

@ Docker/Nginx

mkcert (HTTPS @ localhost)

A simple tool for making locally-trusted development certificates. It requires no configuration. Worthless @ Windows 10.

@ Windows/WSL (Golang/cURL)

Add a self-signed CA (root-ca.crt) to Windows 10.

FAIL:
FIX:
  1. Generate both a CA root certificate and the server's certificate using openssl
    • Use Makefile recipes @ uqrate/v1/assets/keys/tls/openssl/
  2. Per OS:
    • @ Linux

      • Either append the content of the CA's root certificate file (root-ca.crt) to ca-certificates.crt file of the /etc/ssl/certs/ directory, or copy the file itself to that directory.

        sudo vim /etc/ssl/certs/ca-certificates.crt
        • @ XPC, root-ca-swarm.foo.crt is a standalone file under /etc/ssl/certs.
        • @ HTPC, the content of that certificate is appended to ca-certificates.crt file.
    • @ Windows
      • Certificates.msc : Local Machine or Current User
        • Note that WSL apps don't use Windows certs store @ HTPC configuration, Adding CA (openssl) @ Current User failed to fix either Golang or cURL failure. Then, added to Local Machine, but that failed to fix issue @ cURL (Golang untested). Then appended to WSL's certs file (above method), and that fixed at both cURL and Golang. BTW, "Golang" means go run ...; was reporting x509 Certificate Signed by Unknown Authority.
        • Open per Win+R > MMC > File > Add/Remove Snap-in > Certificates > Trusted Root Certification Authorities > All Tasks > Import
          • Import: root-ca.crt

        • @ PowerShell; alt method; effectiveness is unknown.

          Import-Certificate -FilePath "root-ca.crt" -CertStoreLocation Cert:\LocalMachine\Root
      • @ Firefox, mod required to enable the newly added root certificate

acme.sh

TL;DR

HowTo

domain='target.tld'
len='4096'    # @ RSA
len='ec-384'  # @ Elliptic Curve
AWS_ACCESS_KEY_ID="AKI..."
AWS_SECRET_ACCESS_KEY="58X...mdS"

acme.sh --issue --server letsencrypt --dns dns_aws --ocsp-must-staple --keylength $len -d "$domain" -d "*.$domain"

Certbot | certbot @ Docker | certbot-dns-route53 | +DNS Provider plugin

Certbot is the client advised by letsencrypt.org; we used acme.sh instead only because its Docker image was better advertised. Certbot @ docker should perform as well. Both automate the cert request and the DNS-01 challenge per suitable DNS providers (both handle AWS/Route53)

The dns_route53 plugin automates the process of completing a dns-01 challenge (DNS01) by creating, and subsequently removing, TXT records using the Amazon Web Services Route 53 API.

go-acme/lego | Docs

Let's Encrypt client and ACME library written in Go.

Integrations w/ DNS providers; AWS Route53

CertMagic (Caddy)

mature, robust, and capable ACME client integration for Go

// @ HTTP
http.ListenAndServe(":80", mux)

//... replace that with this ...

// @ CertMagic : HTTPS with HTTP->HTTPS redirects 
certmagic.HTTPS([]string{"example.com"}, mux)

Compare SSL Certificates

EV Certificates

Let's Encrypt : ACME protocol : Client implementations (Golang)

The "Let’s Encrypt" organization is a CA. In order to get a certificate for your website’s domain from them, you have to demonstrate control over the domain. This is accomplished using the ACME protocol.

ZeroSSL Pricing

No EV (Extended Validation) certificates.

Domain Validation (DV) certificates ONLY.

@ Free …

    90-Day Certificates - 3
    90-Day ACME Certs   - unlimited

@ $100/yr …

    90-Day Certificates - unlimited
    1-Year Certificates - 3
    Multi-Domain Certs
    REST API Access
    90-Day ACME Certs   - unlimited