Upgrading from 1.36.0 to 1.37.1

Introduction

My homelab runs on a self-hosted Kubernetes cluster deployed with my own flavour of k8s on Fedora CoreOS nodes. This post walks through how I upgraded it from 1.36.0 to 1.37.1, one node at a time.

I don’t think anything is preventing me from upgrading straight to 1.37.1 but I know kubespray probably got more battle-tested on the latest patch so as a precaution I’ll run the update in two steps:

1. 1.36.0 to the latest 1.36.x patch (1.36.5)
2. 1.36.5 to 1.37.1

Before touching the cluster, I also had to get my project’s fork in order. My changes live on a homelab branch that I wanted to rebase on top of upstream master, so the first half of this post is the git work and the second half is the actual upgrade.

Preparing the repository

Pulling the latest upstream changes

I start on master and stash anything untracked so it doesn’t get in the way of my git pull:

git switch master

# stash untracked files
git stash push --include-untracked -m "exclude new files"

# pull latest updates from the origin
git fetch
git pull

Squashing the homelab branch

My homelab branch had accumulated a lot of small commits. I want my commit history to look cleaner so a single big commit instead of 250 smaller ones is the way to go for me

git switch homelab
git reset --soft 28bdeb8583c29b001afbca84bd27259a935eeaed

# stage and commit the squash
git add -A
git commit -m "squash v3"

Rebasing on master

Now the rebase itself:

git rebase master

Kubespray changes a lot between releases, and my customizations touch files that upstream also changed, so conflicts are expected. I want my homelab version to win on any conflicting hunk. That is what the -X theirs strategy option does, but the naming is confusing during a rebase:

• In a merge, ours is the branch you’re on and theirs is the branch you merge in.
• In a rebase, it’s reversed. ours is the branch you’re rebasing onto (master) and theirs is your commits being replayed.

So during a rebase, -X theirs means “prefer my homelab changes”:

# prefer my homelab changes on any conflicting hunk
git rebase -X theirs master
# or specific commit 
git rebase -X theirs 28bdeb8583c29b001afbca84bd27259a935eeaed

I first tried it against the specific commit hash, but what I ended up running was against whatever the latest commit in master is. I would have rather rebased on top of a tagged release but there was no release yet for 1.37 by the time I wanted to update.

Keep in mind that this only resolves conflicting hunks automatically. It doesn’t mean upstream changes in master that don’t conflict are dropped, so it’s still worth looking over the diff afterwards.

Pushing the result

Since the rebase rewrote history, the push has to be forced:

git push source homelab --force

I keep this repository mirrored on both my self-hosted GitLab and GitHub. If you need to recreate that setup, one remote called all with two push URLs does it:

git remote add all git@gitlab.thekor.eu:kube/kubespray.git

git config --global user.name "Hugo Pratt"
git config --global user.email "you@example.com"

git remote set-url --add --push all git@gitlab.thekor.eu:kube/kubespray.git
git remote set-url --add --push all git@github.com:hupratt/kubespray.git

git push all homelab --force
git branch --set-upstream-to=all/homelab homelab

A safer alternative to --force is --force-with-lease, which refuses to overwrite the remote if someone pushed something you haven’t seen. On a repo that only I push to it makes little difference, but it’s good to know.

Upgrading the control plane

Make sure you’re on the right cluster

Before running anything destructive, I confirm which context kubectl is pointing at:

kubectl config get-contexts
kubectl config use-context homelab-prod-context

The workflow per node

My control plane has three nodes: coreos-cp-1, coreos-cp-2 and coreos-cp-3. I upgrade them strictly one after the other, and every node goes through the same four steps.

1. Drain the node so its pods move to the other nodes:

kubectl drain coreos-cp-1 --ignore-daemonsets --delete-emptydir-data

2. Gather facts. Kubespray relies on cached facts about every host. When you restrict a run with --limit, the other hosts’ facts have to be available already, otherwise the playbook can fail on missing variables:

ansible-playbook -i inventory/homelab-prod/inventory.ini --become cluster.yml -t gather-facts

3. Upgrade to the latest patch of the current minor (1.36.5), then verify the node is healthy:

ansible-playbook upgrade-cluster.yml -b -i inventory/homelab-prod/inventory.ini -e kube_version=1.36.5 --limit "coreos-cp-1"

Kubespray automatically uncordons your node as soon as it successfully finishes so you can verify that rook-ceph and the node is healthy right about now.

During the update you can verify that the ceph operator set the OSD to no-out to prevent it from automatically recreating the OSD somewhere else. This is certainly useful when a disk goes bad but it would be a negative side effect for an update procedure since it’s just temporary.

bash-5.1$ ceph -s
  cluster:
    id:     876327be-c95c-448e-a08b-b2fb4c8ecb7e
    health: HEALTH_WARN
            1/3 mons down, quorum t,z
            1 osds down
            1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
            1 host (1 osds) down
            Degraded data redundancy: 199910/599730 objects degraded (33.333%), 81 pgs degraded, 81 pgs undersized
 
  services:
    mon: 3 daemons, quorum t,z (age 15m) [leader: t], out of quorum: w
    mgr: a(active, since 2h), standbys: b
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 2 up (since 15m), 3 in (since 93m)
 
  data:
    volumes: 1/1 healthy
    pools:   4 pools, 81 pgs
    objects: 199.91k objects, 236 GiB
    usage:   713 GiB used, 4.8 TiB / 5.5 TiB avail
    pgs:     199910/599730 objects degraded (33.333%)
             81 active+undersized+degraded
 
  io:
    client:   7.6 KiB/s rd, 336 KiB/s wr, 2 op/s rd, 11 op/s wr
 
  progress:
    Global Recovery Event (0s)
      [............................] 
 

4. Gather facts, drain the node again and upgrade to the new minor (1.37.1):

ansible-playbook -i inventory/homelab-prod/inventory.ini --become cluster.yml -t gather-facts
kubectl drain coreos-cp-1 --ignore-daemonsets --delete-emptydir-data
ansible-playbook upgrade-cluster.yml -b -i inventory/homelab-prod/inventory.ini -e kube_version=1.37.1 --limit "coreos-cp-1"

Then I repeat steps 1 to 4 for coreos-cp-2, coreos-cp-3 and coreos-wk-4, swapping the node name in the drain and --limit arguments. Between nodes I check that the node I just upgraded reports the new version and is Ready before moving on:

kubectl get nodes

A PodDisruptionBudget that blocks the drain

The coreos-cp-1 drain hung. The cause was a PodDisruptionBudget on my InfluxDB instance in the monitoring namespace. It requires at least one available pod, but InfluxDB runs as a single replica, so evicting that pod would always violate the budget and kubectl drain waits forever.

For the duration of the upgrade I deleted the PDB:

kubectl delete pdb influxdb-influxdb2 -n monitoring

The remaining nodes drained without a problem except for the local node that has a CNPG cluster and 1 replica for immich. I got around the issue by disabling the PDB and re-enabling it afterwards:

# disable PDB
kubectl patch cluster immich-pg -n immich --type merge -p '{"spec":{"enablePDB":false}}'

Cleaning up

Once all three control plane nodes were on 1.37.1, I recreated the PDB I had removed:

kubectl apply -f - <<'EOF'
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: influxdb-influxdb2
  namespace: monitoring
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: influxdb
      app.kubernetes.io/name: influxdb2
EOF

kubectl patch cluster immich-pg -n immich --type merge -p '{"spec":{"enablePDB":true}}'

Lessons learned

I did a similar procedure a couple of months ago in this article but I wasn’t happy with it because I didn’t document all of the CLI commands and ceph had lost its quorum which needed some manual fixes.

This time I experienced a lot less downtime but I still need to figure out why it made my services unavailable. Is this perhaps due to disk I/O and memory pressure? Perhaps every deployment should have at least 2+ replicas? Adding a second worker node would allow me to get two CNPG cluster instances and avoid disruption there as well. Perhaps my readiness probes are not working as expected which would also explain the lag? Since most pods are running on ceph PVCs the storage should already be replicated so the downtime can perhaps be attributed to the time it takes to copy over the contents of the RAM to the other VM.

Managing my own kubernetes distro is full of surprises but that’s the whole point of it. Understanding what those other distros bring out of the box can help me debug a lot more effectively in case an openshift cluster or an RKE2 cluster starts throwing errors.

See you on the next one,

Cheers