K3s

TL;DR

K3s is an unconventional though very useful distro, and has excellent documentation. It installs quickly with zero configuration for its sensible defaults. It includes CRI, CNI, service mesh, load balancer, and ingress controller, all chosen for their light weight, making it the go-to distro for edge devices.

Includes k3s CLI, which functions as some kind of wrapper for some or all of the related tools to perform a variety of common cluster-administration tasks.

The one gotcha for newcomers is that using it requires root access, yet it installs to /usr/local/bin, which is not in the sudoers PATH of many Linux distros.

Overview

K3s is a lightweight "batteries included" Kubernetes distribution created by Rancher Labs, and is a CNCF project. K3s is highly available and production-ready. It has a very small binary size and very low resource requirements. — traefiklabs.io

Architecture

The K3s binary includes components such as the core, which are normally static pods, CNI pods (Flannel by default), and an embedded SQLite via Kine shim (vs etcd), making the system more lightweight. Networking is managed internally by K3s (a bundled Flannel by default). So all pods, including what would be static pods in other distributions, are on the Pod network.

This architecture is a major reason why K3s is so lightweight and popular for edge computing, IoT, and other resource-constrained environments.

k3s CLI

k3s             # List commands
k3s server -h   # List all server config/options
sudo k3s server # Launch a cluster (server node)

# Check host config for k3s
sudo k3s check-config

# Cluster access : kubectl
sudo k3s kubectl 
# Else configure shell
alias k='sudo k3s kubectl'
# Else configure shell for regular user, protecting existing kubeconfig
export KUBECONFIG=~/.kube/k3s_config
sudo cp /etc/rancher/k3s/k3s.yaml $KUBECONFIG &&
    chown $USER:$USER $KUBECONFIG && {
        type -t k || alias k='k3s kubectl'
    }

# To merge multiple kubeconfig (contexts)
export KUBECONFIG=$pathConf1:$pathConf2:$pathConf3
# To save that all as a single kubeconfig file:
kubectl config view --flatten |tee /path/to/new/merged/kubeconfig

# Access K8s API server 
k get node,deploy,ds,sts,svc,ingress
k top node
k top pod -A

# - GET /healthz
# Set cluster server URL : See `k config view` else `k get node -o wide` else `k get svc -A`
name=default # config.clusters[].cluster.name
url="$(k config view -o jsonpath='{.clusters[?(@.name=="'$name'")].cluster.server}')"
tkn="$(k -n default create token default --duration=10m)" # Create/use its token
curl -k -H "Authorization: Bearer $(k -n kube-system create token default)" $url/healthz?verbose

List command options:

k3s server --help

Configuration files:

@ systemd

# Start/Stop @ systemd
# - Servers
sudo systemctl $verb k3s
# - Agents 
sudo systemctl $verb k3s-agent

Quick Start

Requirements

@ RHEL 9

# Configure host
sudo su
systemctl disable firewalld --now
sudo setenforce 0 
sudo sed -i -e 's/^SELINUX=disabled/SELINUX=permissive/' /etc/selinux/config
sudo sed -i -e 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
[[ $(systemctl list-unit-files |grep nm-cloud) ]] && 
    systemctl disable nm-cloud-setup.service nm-cloud-setup.timer

reboot

Installation : k3s-install.sh

By default, values present in a YAML file located at /etc/rancher/k3s/config.yaml will be used on install.

write-kubeconfig-mode: "0644"
tls-san:
  - "foo.local"
node-label:
  - "foo=bar"
  - "something=amazing"
cluster-init: true

Script method

Runs as systemd service : k3s.service

# Install 
curl -sfL https://get.k3s.io |sh - &&
    sudo chmod 0644 /etc/rancher/k3s/k3s.yaml

# Combination of environment variables and flags:
curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" INSTALL_K3S_EXEC="server" sh -s - --flannel-backend none

# Configure for root user
path=/usr/local/bin
[[ $(sudo -i env |grep 'PATH=' |grep $path) ]] || 
    [[ $(sudo cat /root/.bashrc |grep 'PATH=' |grep $path) ]] ||
        echo 'export PATH=$PATH:'"$path" |sudo tee -a /root/.bashrc

Binary method

Runs as terminal session (blocking), so only for quick tests

# Select release : https://github.com/k3s-io/k3s/releases
v=v1.31.3 # K8s version available at K3s

# Install k3s CLI if needed
[[ "$(type -t k3s && k3s --version |grep $v)" ]] ||
    curl -sSLO https://github.com/k3s-io/k3s/releases/download/$v%2Bk3s1/k3s

# Init cluster @ terminal (blocks)
type -t kx && kx |grep k3s ||
    sudo k3s server \
        --write-kubeconfig-mode "0644" \
        --node-label "k3s=true" \
        --cluster-cidr '10.42.0.0/16' \
        --service-cidr '10.43.0.0/16' \
        --cluster-init

Teardown

Use the k3s-killall.sh script to stop all of the K3s containers and reset the containerd state. It cleans up containers, K3s directories, and networking components while also removing the iptables chain with all the associated rules.

The cluster data will not be deleted.

/usr/local/bin/k3s-killall.sh

Uninstall

Uninstalling K3s deletes the local cluster data, configuration, and all of the scripts and CLI tools.

# Server (Control) teardown
/usr/local/bin/k3s-uninstall.sh
# Agent (Worker) teardown
/usr/local/bin/k3s-agent-uninstall.sh

Advanced Configuration

In general, CLI arguments map to their respective YAML key, with repeatable CLI arguments being represented as YAML lists. Boolean flags are represented as true or false in the YAML file.

Equivalent syntaxes:

K3S_KUBECONFIG_MODE="644" k3s server
# OR
k3s server --write-kubeconfig-mode=644

Air-gap Install

  1. Configure host
  2. Install containerd
  3. Load images of a K3s release into private registry

Preliminaries requiring internet access

# Select release : https://github.com/k3s-io/k3s/releases
v=v1.31.3 # K8s version available at K3s

# Pull K3s-images archive
tarball=k3s-airgap-images-amd64.tar.gz
curl -sSLO https://github.com/k3s-io/k3s/releases/download/${v}%2Bk3s1/$tarball

# Load all images (later/elsewhere) into local cache 
# for subsequent tag/push to private registry of target air-gap environment.
type -t docker &&
    docker load -i $tarball

# Pull k3s CLI
curl -sSLO https://github.com/k3s-io/k3s/releases/download/${v}%2Bk3s1/k3s

HA K3s

Usage

sudo k3s kubectl 
# OR
alias k='sudo k3s kubectl'

K3s installs ServiceLB as Service traefic of type LoadBalancer to provide a "public" entrypoint to the cluster; EXTERNAL-IP address.

$ k -n kube-system get svc traefik
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE
traefik   LoadBalancer   10.43.77.118   172.27.240.169   80:32083/TCP,443:30823/TCP   128m

$ ip=$(k -n kube-system get svc traefik -o jsonpath='{.status.loadBalancer.ingress[].ip}')


curl -ki https://172.27.240.169:6443/version/

HTTP/2 401
audit-id: 955d99ca-9427-41b1-9e7d-c1377f630d00
cache-control: no-cache, private
content-type: application/json
...


{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

Authenticate by Bearer token created of a ServiceAccount:

$ k -n kube-system get sa
NAME                                          SECRETS   AGE
...
default                                       0         150m
...

$ k -n kube-system create token default

Adding that raw token string (TOKEN) at "Authorization: Bearer TOKEN" request-header value:

ip=$(k -n kube-system get svc traefik -o jsonpath='{.status.loadBalancer.ingress[].ip}')

curl -k -H "Authorization: Bearer $(k -n kube-system create token default)" https://$ip:6443/healthz?verbose

[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-apiserver-admission-initializer ok
...
healthz check passed

Authenticate using certificate and key of kubeconfig.

alias k='sudo k3s kubectl'
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

ip=$(k -n kube-system get svc traefik -o jsonpath='{.status.loadBalancer.ingress[].ip}')

k config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' \
    |base64 -d \
    |tee client.crt

k config view --raw -o jsonpath='{.users[0].user.client-key-data}' \
    |base64 -d \
    |tee client.key

curl -k --cert client.crt \
        --key client.key \
        https://$ip:6443/healthz?verbose

# OR, all in one:
curl -k \
    --cert <(k config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' |base64 -d) \
    --key <(k config view --raw -o jsonpath='{.users[0].user.client-key-data}' |base64 -d) \
    https://$(k -n kube-system get svc traefik -o jsonpath='{.status.loadBalancer.ingress[].ip}'):6443/healthz?verbose

@ /healthz?verbose

[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-apiserver-admission-initializer ok
...
healthz check passed

@ /version

{
  "major": "1",
  "minor": "30",
  "gitVersion": "v1.30.3+k3s1",
  "gitCommit": "f646604010affc6a1d3233a8a0870bca46bf80cf",
  "gitTreeState": "clean",
  "buildDate": "2024-07-31T20:30:52Z",
  "goVersion": "go1.22.5",
  "compiler": "gc",
  "platform": "linux/amd64"
}

Network : iperf3

☩ k get node -o wide
NAME   STATUS   ROLES                  AGE     VERSION        INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION                       CONTAINER-RUNTIME
xpc    Ready    control-plane,master   4h40m   v1.31.3+k3s1   172.27.240.169   <none>        Ubuntu 22.04.4 LTS   5.15.167.4-microsoft-standard-WSL2   containerd://1.7.23-k3s2

☩ k run nbox-s --image=docker.io/n^Colaka/netshoot --restart=Never  -- iperf3 -c

☩ k get pod -o wide
NAME     READY   STATUS    RESTARTS   AGE   IP           NODE   NOMINATED NODE   READINESS GATES
nbox-s   1/1     Running   0          19s   10.42.0.23   xpc    <none>           <none>

☩ k run nbox-c --image=docker.io/nicolaka/netshoot -it --rm -- iperf3 -c 10.42.0.23
If you don't see a command prompt, try pressing enter.
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.03 GBytes  34.6 Gbits/sec  141   1.00 MBytes
[  5]   1.00-2.00   sec  3.94 GBytes  33.8 Gbits/sec    2   1.00 MBytes
[  5]   2.00-3.00   sec  3.88 GBytes  33.4 Gbits/sec    0   1.00 MBytes
[  5]   3.00-4.00   sec  3.81 GBytes  32.7 Gbits/sec    0   1.00 MBytes
[  5]   4.00-5.00   sec  3.73 GBytes  32.1 Gbits/sec    0   1.00 MBytes
[  5]   5.00-6.00   sec  3.74 GBytes  32.1 Gbits/sec    0   1.00 MBytes
[  5]   6.00-7.00   sec  3.79 GBytes  32.5 Gbits/sec    0   1.00 MBytes
[  5]   7.00-8.00   sec  3.79 GBytes  32.6 Gbits/sec    0   1.00 MBytes
[  5]   8.00-9.00   sec  3.76 GBytes  32.3 Gbits/sec    0   1.00 MBytes
[  5]   9.00-10.00  sec  3.88 GBytes  33.3 Gbits/sec    0   1.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  38.5 GBytes  33.1 Gbits/sec  143             sender
[  5]   0.00-10.00  sec  38.5 GBytes  33.1 Gbits/sec                  receiver

iperf Done.
pod "nbox-c" deleted