Kubernetes Cluster

Kubernetes Cluster

Kubernetes Cluster

Kubernetes (k8s) Cluster with Control Plane & Worker Nodes

1. Prerequisites

Kubernetes' requirements are as follows:

RAM: >2GB for each node
CPU: >2 core (Control plane)
Unique hostname for each node
Firewall ports open

2. Disable swap

You must disable swap:

Copy

swapoff -a

Also, edit /etc/fstab, and comment out the line that contains swap. For example:

Copy

# COMMENT THE LINE THAT CONTAINS 'swap'
#                 vvvv
#/dev/sda3  none  swap  sw  0 0

3. Configure Kernel

It's necessary to modify some Kernel settings.

Copy

# Load modules
sudo modprobe overlay
sudo modprobe br_netfilter

# Permanently save changes
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

# Tweak sysctl
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
# enables bridged traffic to pass through iptables,
# crucial for routing between nodes and pods
net.bridge.bridge-nf-call-iptables = 1
# allows IP forwarding for pod-to-pod communication across node interfaces
net.ipv4.ip_forward = 1
# manages IPv6 traffic on bridged interfaces through ip6tables, important for IPv6 networking environments
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# Load sysctl tweaks
sudo sysctl --system

4. Configure firewall

Make sure to open the following ports:

Control Plane:
- 6443 (TCP): Kubernetes API Server
- 2379-2380 (TCP): etc server client API
- 10250 (TCP): Kubelet API
- 10257 (TCP): kube-controller-manager
- 10259 (TCP): kube-scheduler
Worker Nodes:
- 10250 (TCP): Kubelet API
- 10256 (TCP): kube-proxy
- 30000-32767 (TCP/UDP): NodePort Services

CentOS/Fedora/RHEL (firewalld):

Copy

# Control Plane
firewall-cmd --zone=public --add-port=6443/tcp --permanent
firewall-cmd --zone=public --add-port=2379-2380/tcp --permanent
firewall-cmd --zone=public --add-port=10250/tcp --permanent
firewall-cmd --zone=public --add-port=10257/tcp --permanent
firewall-cmd --zone=public --add-port=10259/tcp --permanent
firewall-cmd --reload

# Worker Nodes
firewall-cmd --zone=public --add-port=10250/tcp --permanent
firewall-cmd --zone=public --add-port=10256/tcp --permanent
firewall-cmd --zone=public --add-port=30000-32767/tcp --permanent
firewall-cmd --zone=public --add-port=30000-32767/udp --permanent
firewall-cmd --reload

Debian (ufw):

Copy

# Control Plane
ufw allow 6443,2379:2380,10250,10257,10259/tcp

# Worker Nodes
ufw allow 10250,10256,30000:32767/tcp
ufw allow 30000:32767/udp

Warning

This is only an example config. It is highly recommended you modify it so that only your nodes can communicate on these ports.

5. Install packages

CentOS/Fedora/RHEL:

Copy

# Set SELinux to permissive mode for ease of troubleshooting
sudo setenforce 0
sudo sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/sysconfig/selinux

# Add containerd repo & install it
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf install containerd.io -y

# Add kubernetes repo
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

# Install Kubernetes
sudo dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

Debian:

Copy

# Update repos
sudo apt update

# Install required packages
sudo apt install -y apt-transport-https ca-certificates curl gpg

# Copy Kubernetes gpg key for apt
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# Add Kubernetes repository
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

# Update repos
sudo apt update

# Install Kubernetes & other packages
sudo apt install docker-cli containerd kubectl kubeadm kubelet

Gentoo:

Copy

# Control Plane
doas emerge -a app-containers/{containerd,cri-o} sys-cluster/kube-{apiserver,controller-manager,proxy,scheduler} sys-cluster/kube{adm,ctl,let,letctl}

# Worker Nodes
sudo emerge -a app-containers/{containerd,cri-o} sys-cluster/kube{let,-proxy}

6. Configure packages

Copy

# Containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

# Gentoo users: do not run if you don't use systemd
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
sudo systemctl restart containerd

# Configure crictl (cri-o)
cat <<EOF | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 3
EOF

7. Start services

Copy

# Systemd users (Debian, CentOS, Fedora, RHEL, ...)
sudo systemctl enable --now containerd
sudo systemctl enable --now kubelet

# Gentoo users (OpenRC)
sudo rc-update add containerd default
sudo rc-update add kubelet default
sudo rc-service containerd start
# Note: do not run the line below on the control plane.
# If the control plane also acts as a worker (2-node cluster),
# enable the 'kubelet' service only after creating the cluster.
sudo rc-service kubelet start

8. Create cluster

Create a cluster with IP addresses in the range 10.88.0.0/16:

Copy

sudo kubeadm init --pod-network-cidr 10.88.0.0/16

Info

If any problems arise, reset & delete the cluster with all Kubernetes data with these commands:

Copy

sudo systemctl stop kubelet
sudo kubeadm reset
sudo rm -rf /etc/kubernetes /var/lib/etc $HOME/.kube
sudo pkill -9 "kube*"

Start kubelet:

Copy

sudo systemctl start kubelet

Once the cluster has been created, you'll see a command like this:

Copy

sudo kubeadm join CONTROL-PLANE-IP:6443 \
--token TOKEN \
--discovery-token-ca-cert-hash sha256:HASH

This command has to be executed on each worker node that you want to add to the cluster. Replace the bold parameters with those retrieved from the command output.

After creating the cluster, copy the configuration into your home directory:

Copy

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Verify the number of nodes in the cluster:

Copy

kubectl get nodes

Example output:

NAME	STATUS	ROLES	AGE	VERSION
ControlPlane	Ready	control-plane	1m	v1.33.1
Worker1	Ready	<none>	1m	v1.33.1
Worker2	Ready	<none>	1m	v1.33.1
Worker3	Ready	<none>	1m	v1.33.1

Info

Each node tipically becomes Ready almost immediately, or in less than 30 seconds from its add/join date.

9. Install Calico CNI

Calico CNI (Container Network Interface) provides network connectivity, security, and observability for containerized applications. Install it with these commands:

Copy

# Setup tigera operator
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/master/manifests/tigera-operator.yaml

# Download custom resources file
curl https://raw.githubusercontent.com/projectcalico/calico/master/manifests/custom-resources.yaml -O

# Replace default network '192.168.0.0/16' with '10.88.0.0/16'
sed -i "s/192.168.0.0\/16/10.88.0.0\/16/" custom-resources.yaml

# Create custom resources
kubectl create -f custom-resources.yaml

10. Test environment

Verify that the cluster, the Control Plane, and the Worker Nodes work correctly.

Create a test namespace:

Copy

kubectl create namespace test

Create a new file nginx.yaml, edit it, and add the following:

Copy

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx
  namespace: test
data:
  nginx.conf: |
    worker_processes auto;

    events {
        worker_connections 1024;
    }

    http {
        include       mime.types;
        default_type  application/octet-stream;

        server {
            listen       80;
            server_name  _;

            location / {
              # Return public IP
              resolver 1.1.1.1;
              resolver_timeout 5s;
              proxy_pass https://ifconfig.me/ip;
            }
        }
    }

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx
  namespace: test
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
          - containerPort: 80
        volumeMounts:
        - name: nginx-config-volume
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
      volumes:
      - name: nginx-config-volume
        configMap:
          name: nginx

---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: test
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 30000
  type: NodePort

Apply the file:

Copy

kubectl apply -f nginx.yaml

Make sure the pods have been started:

Copy

watch kubectl get pods -n test

Run the script below to check that the load balancing functionality works as expected:

Copy

#!/bin/bash
IP=controlplane
PORT=30000

declare -A buffer

for i in {1..10}; do
    ip=$(curl http://$IP:$PORT 2>/dev/null)
    echo "IP: $ip"

    if [[ -v buffer["$ip"] ]]; then
        buffer["$ip"]=$((buffer["$ip"] + 1))
    else
        buffer["$ip"]=1
    fi
done

for ip in "${!buffer[@]}"; do
    seen=${buffer["$ip"]}
    echo "$ip seen $seen times"
done

For example, if the cluster has 2 worker nodes, you should see the following output:

Copy

100.0.10.11 seen 5 times
100.0.10.12 seen 5 times

Info

Note that the ratio may not always be 50% for each node (requests_served / total_requests). This is more than normal, as Kubernetes takes into account the system's load, as well as running round-robin scheduling.

This means that, for example, you could see:

Copy

100.0.10.11 seen 4 times
100.0.10.12 seen 6 times

...and immediately after:

Copy

100.0.10.11 seen 6 times
100.0.10.12 seen 4 times

Remove the test environment:

Copy

kubectl delete -n test configmap/nginx daemonset.apps/nginx service/nginx
kubectl delete namespace test

← Back to the main page