InfrastructureAWSGoDatabaseMonitoringRaspberryPIPythonTerraformGitAIDIYData Analytics
Catalogue
Building a Home Kubernetes Cluster
Posted 2022-05-06RaspberryPI18 minutes read (About 2728 words)

Building a Home Kubernetes Cluster

🌐 日本語で読む

ToC

I built a Kubernetes cluster on Raspberry Pi.

Thanks to my company’s “tech support program,” the Raspberry Pi became an eligible item, and I was able to spend money on it without reservation.
Riding on that good fortune, I took on the challenge of finally raising a k8s cluster at home—an experience that is the very pinnacle of engineering joy.

Normally I work with Fargate on AWS in most cases, so there are no occasions to use Kubernetes. I was curious about it and wanted to broaden my learning.

Goals

The goal is to install an OS on a bare-metal environment like the Raspberry Pi, build the Kubernetes-related packages, and understand the overall flow required for the setup.

I’ll go through the following:

  1. Installing the Raspberry Pi OS
  2. Building the Kubernetes cluster
  3. Setting up MetalLB

There were a lot of things to learn and plenty of pitfalls, so I’d like to write them down below.

List of items I purchased

In late April 2022, the Raspberry Pi was more commonly sold as part of a starter kit rather than as a standalone unit.
A starter kit is somewhat more expensive than the standalone unit, but for that part I made full use of the support program from my company ♪

Writing the OS with Raspberry Pi Imager

I selected Raspberry Pi OS Lite (32-bit) Bullseye, which was the latest as of 2022-04-26.
This time a GUI was not needed for the requirements, and I wanted to use as lightweight an image as possible.

Configuring the hostname and Wi-Fi settings here makes things easier later.

Write the image to the SD card, insert it into the Raspberry Pi, and boot it up.

Enabling cgroups

To use Docker, enable cgroups.

1
2
3
$ sudo nano /boot/cmdline.txt

cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

nano and vi were already installed, but if you want to use vim, see this article.

The following was helpful for understanding cgroups.

Kubernetes で cgroup がどう利用されているか - VA Linux エンジニアブログ
cgroup とは、Linux カーネルの機能の1つであり、プロセスやスレッドが利用するリソースの制限や分離を行うための機能で、Linux コンテナの根幹を成す技術の1つでもあります。本記事では、Kubernetes のいくつかの機能を例に挙げ、cgroup がどう利用されている…

Disabling swap

1
2
3
$ sudo swapoff --all
$ sudo systemctl stop dphys-swapfile
$ sudo systemctl disable dphys-swapfile

The reason for disabling swap is mentioned in the official documentation.

https://kubernetes.io/ja/docs/setup/production-environment/tools/kubeadm/_print/#始める前に

Swap must be off. For the kubelet to work properly, swap must always be disabled.

Fixing the IP address

Reserving a fixed IP on the router makes things easier, since it won’t change on reboot.

For the approach of fixing the IP on the Raspberry Pi side by editing /etc/dhcpcd.conf instead of on the router, the following is helpful.
How to assign a static IP address to a Raspberry Pi

Reboot once here to apply everything.

1
$ sudo reboot

Installing Docker

I installed it following Installing a CRI.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// Fetch the public key needed when updating the package repository information. Without it, a GPG error occurs.
$ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

// Register the stable Docker repository for armhf debian
$ echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list

// Update the repository
$ sudo apt-get update

// Install Docker
$ sudo apt-get -y install docker-ce docker-ce-cli containerd.io
$ sudo systemctl enable docker

// Add the pi user to the docker group so it can operate docker
$ sudo usermod pi -aG docker

Confirming that the Cgroup Driver is set to use systemd

1
2
3
4
$ sudo docker info | grep Cgroup

 Cgroup Driver: systemd
 Cgroup Version: 2

The official docs recommend using systemd, as below.

Changing the settings so that your container runtime and kubelet use systemd as the cgroup driver stabilized the system. Note the native.cgroupdriver=systemd option in the Docker configuration below.

Installing kubeadm

I ran the following steps along with the official docs.

Letting iptables see bridged traffic

See the official documentation

Check whether br_netfilter is loaded.

1
2
3
4
5
$ lsmod | grep br_netfilter

br_netfilter           32768  0
bridge                180224  1 br_netfilter
ipv6                  520192  28 br_netfilter,bridge

If nothing is displayed, br_netfilter is not loaded, so run the following to load it explicitly.

1
$ modprobe br_netfilter
1
2
3
4
5
$ cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
$ sudo sysctl --system

Ensuring iptables does not use the nftables backend

Official docs

The nftables backend is not compatible with the current kubeadm packages (because it duplicates firewall rules and breaks kube-proxy).

As the official explanation says, because using nftables for iptables can prevent Kubernetes from working correctly, I switch iptables to the legacy version.

1
2
3
4
5
6
7
8
// Make sure the legacy binaries are installed
$ sudo apt-get install -y iptables arptables ebtables

// Switch to the legacy versions.
$ sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
$ sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
$ sudo update-alternatives --set arptables /usr/sbin/arptables-legacy
$ sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy

Reference: Introduction to nftables

Installing kubeadm, kubelet, and kubectl

Finally, it’s time to install kubeadm.

Official docs

With version 1.23.6, the latest as of 2022/04/30, I hit an error where the kubelet failed to start, so
I chose the 1.22 series for the version.

1
2
3
4
5
6
7
8
9
10
11
12
$ sudo apt-get update && sudo apt-get install -y apt-transport-https curl
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
$ cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
$ sudo apt-get update

// Install the 1.22 series
$ sudo apt-get install -y kubelet=1.22.7-00 kubeadm=1.22.7-00 kubectl=1.22.7-00

// Pin the versions
$ sudo apt-mark hold kubelet kubeadm kubectl

Building the Kubernetes cluster

Official docs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
// Specify 10.244.0.0/16 in order to set up flannel during cluster initialization
// see: https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16

...
// Copy the last line
kubeadm join <master node ip>:6443 --token yyyy \
        --discovery-token-ca-cert-hash sha256:xxxxxxxx

// The cluster startup settings that are output during the above creation
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

// Set up flannel
$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

// You can watch it start up ♪
$ kubectl get pod --all-namespaces

NAMESPACE     NAME                               READY   STATUS     RESTARTS   AGE
kube-system   coredns-78fcd69978-sv52p           0/1     Pending    0          111s
kube-system   coredns-78fcd69978-t5glm           0/1     Pending    0          111s
kube-system   etcd-pikube01                      1/1     Running    0          2m
kube-system   kube-apiserver-pikube01            1/1     Running    0          2m4s
kube-system   kube-controller-manager-pikube01   1/1     Running    0          2m
kube-system   kube-flannel-ds-w2bqt              0/1     Init:0/2   0          9s
kube-system   kube-proxy-kpm8w                   1/1     Running    0          111s
kube-system   kube-scheduler-pikube01            1/1     Running    0          2m

flannel is useful for building networking such as inter-container connectivity, and it pairs well with k8s.

Registering worker nodes

Run the command that was output when the cluster was created on the master node.
Run the following on the worker nodes.

1
2
3
4
5
6
7
8
9
$ sudo kubeadm join <master node ip>:6443 --token xxx \
        --discovery-token-ca-cert-hash sha256:yyy

...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

Note that the token is time-limited.

You can check the token’s expiration on the master node.

1
2
3
4
master$ kubeadm token list

TOKEN                     TTL         EXPIRES                USAGES                   DESCRIPTION                                                EXTRA GROUPS
xxx   23h         2022-04-28T13:12:39Z   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token

When the token has expired

Reissue it on the master node.
This reissues the token and also outputs the command for running kubeadm join on the worker nodes.

1
master$ kubeadm token create --print-join-command

Checking whether the worker nodes are registered in the cluster

1
2
3
4
5
6
master$ kubectl get nodes

NAME       STATUS     ROLES                  AGE   VERSION
pikube01   Ready      control-plane,master   32h   v1.22.7
pikube02   Ready      <none>                 32m   v1.22.7
pikube03   NotReady   <none>                 18s   v1.22.7

Adding labels

1
2
master$ kubectl label node pikube02  node-role.kubernetes.io/worker=
master$ kubectl label node pikube03  node-role.kubernetes.io/worker=

Listing the nodes again, you can see that ROLES has been labeled.

1
2
3
4
5
6
$ kubectl get nodes

NAME       STATUS   ROLES                  AGE    VERSION
pikube01   Ready    control-plane,master   32h    v1.22.7
pikube02   Ready    worker                 39m    v1.22.7
pikube03   Ready    worker                 7m8s   v1.22.7

Operating the cluster with kubectl from a local machine

1
2
3
4
5
6
7
// Copy the output
master$ kubectl config view --raw

macOS$ vi ~/.kube/config
// Paste the copied content above and save

macOS$ kubectl get nodes

Installing MetalLB

References:

1
2
3
// Enable IPv4 packet forwarding on all interfaces
$ sudo sysctl net.ipv4.conf.all.forwarding=1
$ sudo iptables -P FORWARD ACCEPT

Proceed according to the configuration in MetalLB > Installation.

1
2
3
4
5
6
7
8
9
10
11
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/namespace.yaml
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/metallb.yaml

// Confirm that the metallb-related pods are running
$ kubectl get -n metallb-system pods

NAME                          READY   STATUS    RESTARTS   AGE
controller-66445f859d-qg8cz   1/1     Running   0          30s
speaker-bzzzc                 1/1     Running   0          30s
speaker-vbhdf                 1/1     Running   0          30s
speaker-vslj8                 1/1     Running   0          30s

For addresses, 192.168.11.200-192.168.11.220 specifies a range that can be obtained via DHCP.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// Start metallb in layer2 mode
$ cat <EOF> metallb-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
      - name: default
        protocol: layer2
        addresses:
          - 192.168.11.200-192.168.11.220
EOF

$ kubectl apply -f metallb-config.yaml

Deploy nginx with type: LoadBalancer and confirm that metallb has assigned an IP.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
$ cat <EOF> nginx.deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - port: 80
      targetPort: 80
  type: LoadBalancer
EOF

$ kubectl apply -f nginx.deployment.yml

Confirm that the assigned IP is 192.168.11.200 and that it can be accessed from the outside.

1
2
3
4
5
$ kubectl get svc

NAME            TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)        AGE
kubernetes      ClusterIP      10.96.0.1      <none>           443/TCP        148m
nginx-service   LoadBalancer   10.109.50.62   192.168.11.200   80:31270/TCP   57m
1
2
3
$ curl 192.168.11.200

// "Welcome to nginx!" is displayed

When you cannot access it from machines other than the Raspberry Pi

In my own environment,
I could access the EXTERNAL-IP from the macOS machine where I configured the Raspberry Pi right after nginx started,
but a few minutes later, I ran into an issue where it became inaccessible.

I solved it by referring to the following.

Reference: LoadBalancer using Metallb on bare metal RPI cluster not working after installation

MetalLB layer2 mode does not receive broadcast packets unless promiscuous mode is enabled.
Therefore, by enabling reception of broadcast packets as below,
I was able to confirm connectivity from macOS –> MetalLB.

1
$ sudo ifconfig wlan0 promisc

promisc stands for “promiscuous,” meaning “indiscriminate,” i.e., putting the interface into a mode that reads all traffic.

Since this setting disappears on server reboot, it’s a good idea to set it up in crontab.

1
2
3
4
$ sudo crontab -e

// Add the following to the last line
@reboot sudo ifconfig wlan0 promisc

Wrap-up

The pitfalls were as follows:

  • kubeadm and kubelet didn’t work on the latest 1.23 series
    • Resolved by downgrading by one minor version
  • I couldn’t connect to the External IP emitted by MetalLB
    • Resolved by enabling promiscuous mode

Going forward, I’ll actually build services while focusing on the following ♪

  • CI/CD
  • Monitoring

That’s all.
I hope this is helpful.

References

https://qiita.com/reireias/items/0d87de18f43f27a8ed9b

Building a Home Kubernetes Cluster

https://kenzo0107.github.io/en/2022/05/06/ouchi-kubernetes/

Author

Kenzo Tanaka

Posted on

2022-05-06

Licensed under

Like this article? Support the author with

×