Upgrading a Rook-Ceph Cluster Across Five Releases

Introduction

Ceph upgrades don’t like to be rushed. Jump too many versions at once and the operator refuses to reconcile; skip a step and you can end up with an unhealthy cluster that’s hard to repair. This article walks through my home-prod’s upgrade path — taking a Rook-Ceph deployment from Ceph v18.2.4 all the way to v20.2.4 — along with the operator version hops required to get there safely, and a cleanup pass to fix a HEALTH_WARN caused by insecure cephx key types.

Syncing the Fork with Upstream First

Since this cluster’s manifests live in a forked repo tracking a homelab branch, the first step was squashing my changes into a single commit, pulling in the latest upstream Rook history and tags before touching anything in the cluster:

git remote add upstream https://github.com/rook/rook.git
git fetch upstream
git checkout -b master upstream/master
git push origin --tags

With the remote tracking configured on the working branch:

git branch --set-upstream-to=origin/homelab homelab

…it becomes a one-liner to bring upstream changes into the homelab branch before each upgrade, favoring upstream’s version of any conflicting files:

git fetch upstream
git switch homelab
git rebase -X theirs upstream/master

This keeps the fork current with upstream while preserving homelab-specific changes, and makes the release tags (v1.15.9, v1.16.9, etc.) available locally for the git checkout steps below.

The Upgrade Path

Rook doesn’t support skipping operator versions, so each Ceph major-version bump means walking through several intermediate operator releases first. For this cluster, starting on operator v1.15.3, the path looked like:

v1.15.3 → v1.15.9 → v1.16.9 → v1.17.9 → v1.18.11 (Ceph v18 → v19)
v1.18.11 → v1.19.11 (Ceph v19 → v20) → v1.20.7 

The pattern for every operator hop is the same three steps:

• Check out the target release tag in the local Rook repo, so the matching manifests are on disk:

  git checkout v1.15.9
  

• Apply the manifests for the target release — common.yaml, crds.yaml, and (for later releases) csi-operator.yaml / operator.yaml:

  kubectl apply -f deploy/examples/common.yaml -f deploy/examples/crds.yaml
  

• Bump the operator image:

  kubectl -n rook-ceph set image deploy/rook-ceph-operator \
  rook-ceph-operator=rook/ceph:v1.15.9
  

Repeat all three steps for each intermediate tag (v1.16.9, v1.17.9, v1.18.11, v1.19.11, v1.20.7), letting the operator settle between hops before moving on. The v1.20.7 hop also pulls in two extra manifests:

git checkout v1.20.7
kubectl apply -f deploy/examples/common.yaml -f deploy/examples/crds.yaml \
  -f deploy/examples/csi-operator.yaml -f deploy/examples/operator.yaml
kubectl -n rook-ceph set image deploy/rook-ceph-operator \
  rook-ceph-operator=rook/ceph:v1.20.7

Updating the Ceph’s cluster Version Itself

Once the operator chain has caught up, the actual Ceph version is changed by patching the CephCluster spec:

kubectl -n rook-ceph get cephcluster rook-ceph -o jsonpath='{.spec.cephVersion.image}'
kubectl -n rook-ceph patch cephcluster rook-ceph --type merge \
  -p '{"spec":{"cephVersion":{"image":"quay.io/ceph/ceph:v19.2.3"}}}'

The same pattern applies later for the v20.2.4 jump, once the operator has been walked up to v1.20.7.

Cleaning Up After the Upgrade: Insecure Cephx Key Types

After the version bumps landed, ceph -s reported a HEALTH_WARN with several related complaints:

5 auth client entities with insecure key types
Monitors are configured to allow auth using insecure key types
Monitors are configured to allow creation of insecure key types
4 rotating auth service keys using insecure key types

This comes from CSI client keys still using the older aes cipher rather than aes256k. Fixing it took a few coordinated steps:

1. Refresh the toolbox and CSI controller manager:

kubectl -n rook-ceph set image deployment/rook-ceph-tools \
  rook-ceph-tools=quay.io/ceph/ceph:v20.2.4

kubectl -n rook-ceph rollout restart deployment/ceph-csi-controller-manager

2. Switch on the CSI operator and force key rotation via the CephCluster spec:

kubectl -n rook-ceph patch configmap rook-ceph-operator-config \
  --type merge -p '{"data":{"ROOK_USE_CSI_OPERATOR":"true"}}'

kubectl -n rook-ceph patch cephcluster rook-ceph --type merge -p '
spec:
  security:
    cephx:
      daemon:
        keyRotationPolicy: KeyGeneration
        keyGeneration: 2
'

3. Remove the stale CSI driver objects and force reconciliation so the new drivers pick up regenerated keys:

kubectl delete csidriver \
  rook-ceph.rbd.csi.ceph.com \
  rook-ceph.cephfs.csi.ceph.com

kubectl -n rook-ceph annotate driver rook-ceph.rbd.csi.ceph.com \
  csi.ceph.com/force-reconcile="$(date +%s)" --overwrite

kubectl -n rook-ceph annotate driver rook-ceph.cephfs.csi.ceph.com \
  csi.ceph.com/force-reconcile="$(date +%s)" --overwrite

kubectl rollout restart deploy -n rook-ceph rook-ceph-operator

4. Verify every key is now aes256k:

kubectl -n rook-ceph exec deploy/rook-ceph-tools -- \
  ceph auth dump-keys --format=json |
  jq -r '.data.secrets[] |
    [(.entity.type_str + (if .entity.id != "" then "." + .entity.id else "" end)),
     .auth.key.type_str, .auth.key.created] | @tsv' | column -t

In this run, every daemon and admin key came back aes256k immediately, but the four CSI client keys (csi-cephfs-node, csi-cephfs-provisioner, csi-rbd-node, csi-rbd-provisioner) stuck around as aes and as new .2 suffixed aes256k entries — the rotation had created new keys alongside the old rather than replacing them outright.

5. Once the new keys were confirmed healthy, stop keeping the old ones around:

kubectl -n rook-ceph patch cephcluster rook-ceph --type merge -p '
spec:
  security:
    cephx:
      csi:
        keepPriorKeyCountMax: 0
'

6. Lock the cluster down so no more insecure keys can be generated:

ceph auth del client.rbd-mirror-peer
ceph mon set auth_service_cipher aes256k
ceph config set mon mon_auth_allow_insecure_key false
ceph mon set auth_allowed_ciphers aes256k

7. Confirm the warnings are gone:

ceph mon dump | grep -E 'auth_(allowed|preferred|service)_cipher'
ceph health detail
ceph -s

Result

After the full walk — five operator versions, two Ceph major-version bumps, and a cephx key rotation — the cluster settled at:

health: HEALTH_OK
mon: 3 daemons, quorum t,s,v
mgr: b(active), standby: a
mds: 1/1 up, 1 hot standby
osd: 3/3 up, 3/3 in
pgs: 81 active+clean

Takeaways

• Never skip operator’s minor versions. Rook enforces sequential upgrades for a reason — each step reconciles CRDs and internal state that later steps depend on.
• Apply manifests before bumping the image, every time, for every hop.
• Cephx key rotation isn’t instant or automatic for CSI clients. Expect old and new keys to coexist briefly (keyGeneration bump creates .2 entries) before keepPriorKeyCountMax: 0 retires the originals.
• Lock the cipher policy down last, once you’ve confirmed every client has rotated — disabling aes too early can break clients still mid-rotation.
• Stick to the helm release, despite using the rook operator I should have just stuck to a helm release and have helm manage ceph’s complexity for me.