Shutdown Hook for k8s

Introduction

If you administer Kubernetes you’ve probably hit this moment: you reboot a node, and for a solid 2-3 minutes kubectl get node shows it as NotReady with pods stuck mid-termination, or worse — pods get hard-killed before they’ve had a chance to get recreated onto other nodes gracefully. The node comes back up fine, but the shutdown itself was messy and now you’re potentially stuck with inconsistent databases or storage clusters that refuse to boot.

This article goes into how to do a proper systemd shutdown inhibitor and avoid the messy shutdown.

The naive fix that does not work

The obvious approach is a systemd unit with Before=shutdown.target and an ExecStop= hook that runs kubectl drain on the way down. It’s a pattern you’ll see recommended by a classic sysadmin but it won’t work.

The problem is container runtime internals: every container on the node runs inside its own transient systemd scope (cri-containerd-<id>.scope). When a shutdown transaction starts, all of those scopes get swept up into the general fan-out in parallel — regardless of what ordering you’ve declared for kubelet.service itself. There’s no way to say “drain the node before these scopes stop”

If you try the naive way you’ll see that the Kubernetes API server is already unreachable roughly 20 seconds before kubelet.service had even started stopping. Before=shutdown.target only guarantees your own unit finishes before that target — it does nothing to stop everything else racing toward teardown at the same time.

The fix: hold the whole shutdown hostage

The actual solution is to not let shutdown start at all until the drain is done. systemd-logind supports shutdown inhibitor locks, and a delay-type lock pauses the entire shutdown transition — before any container teardown begins — until the lock is released.

The mechanism:

1. A long-running watcher process holds a systemd-inhibit --what=shutdown --mode=delay lock for the node’s whole uptime.
2. It blocks waiting for the PrepareForShutdown D-Bus signal.
3. When logind broadcasts that signal (because someone ran systemctl reboot or poweroff), the watcher runs the drain script synchronously — cordon, taint, kubectl drain.
4. Only once the drain finishes does the watcher exit and release the lock, letting the actual shutdown proceed.

Because the lock blocks logind itself, nothing downstream — kubelet, container scopes, anything — starts tearing down until the drain is genuinely done. On boot, a companion script waits for the API server to come back, removes the taint, and uncordons the node.

Worth noting: this only fires on a controlled shutdown or reboot that goes through logind. A hard power loss or kill -9 won’t trigger it — there’s no saving those.

Why a separate kubeconfig

These scripts run directly from systemd on the host, not from inside a Pod — so we’ll create a ServiceAccount node-drain-agent together with a token. Each worker node needs its own standing kubeconfig, using a long-lived token tied to a restricted ClusterRole scoped to exactly what cordon/taint/drain/uncordon need. Definitely not the admin kubeconfig.

Two logind gotchas that’ll bite you

InhibitDelayMaxSec is the setting that caps how long any inhibitor — yours or kubelet’s own — can hold up a shutdown. Two things worth knowing before you debug this:

Kubespray templates its own drop-in. If your cluster was provisioned with Kubespray, there’s already a /etc/systemd/logind.conf.d/99-kubelet.conf matching whatever shutdownGracePeriod was set at provisioning time. It’s a static file — the running kubelet process doesn’t rewrite it on restart. So if you hand-edit /var/lib/kubelet/config.yaml later to raise the grace period, this file silently drifts out of sync unless you update it too.

Filename ordering matters more than you’d think. Because 99-kubelet.conf sorts after a plain numeric prefix, a drop-in named something like 90-...conf will lose. Any letter prefix beats any digit prefix in a plain alphabetical sort — so naming your own file zz-k8s-drain-logind.conf guarantees it’s applied last so it wins and the settings defined in it get precedence over any other .conf file.

After installing your drop-in, restart systemd-logind and verify the value actually took:

systemctl restart systemd-logind
busctl get-property org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager InhibitDelayMaxUSec

If it’s still reporting the default (5 seconds), something didn’t apply.

Respecting kubelet’s own shutdown behavior

Kubelet has its own graceful shutdown settings:

shutdownGracePeriod: 150s
shutdownGracePeriodCriticalPods: 20s

Static pods — including the API server itself — are treated as critical and stay alive until the final shutdownGracePeriodCriticalPods window. That leaves roughly 150s - 20s = 130s of a still-live API server for the watcher to actually run the drain in, which is comfortable margin over a kubectl drain timeout of 90 seconds.

It’s worth being clear-eyed that kubelet’s graceful pod termination and the external kubectl drain are two independent mechanisms reacting to the same shutdown signal on their own timelines — the drain isn’t the only thing terminating pods during shutdown. That’s fine; the goal isn’t to be the sole actor, just to make sure nothing gets hard-killed instantly.

Fedora CoreOS specifics

CoreOS’s /usr is read-only, so scripts can’t live at the conventional /usr/local/bin. They need to go under /opt (a writable symlink to /var/opt) or /var/home/core/bin, with the service files’ ExecStart= paths adjusted accordingly.

busctl, needed for catching the D-Bus signal, ships with systemd itself — no extra package required. kubectl, on the other hand, usually isn’t present by default on a Kubespray-built CoreOS worker, so it needs to be dropped onto PATH (e.g. /opt/bin/kubectl) as a static binary matching the cluster’s minor version. Both scripts no-op cleanly if kubectl is missing, so a misconfigured node won’t hang — it just won’t do anything useful.

Verifying it actually works

The real test is a live reboot:

systemctl reboot

you can then watch the drain

journalctl -t k8s-shutdown-watcher -f --no-pager
journalctl -u k8s-node-drain.service -f --no-pager
journalctl -t k8s-drain -f --no-pager

From another node, watch the taint appear and the node cordon:

kubectl get node <node> -w
kubectl describe node <node> | grep -A2 Taints

After it comes back, confirm the taint is gone and the node is schedulable again. And after the fact, the previous boot’s logs tell the real story:

journalctl -t k8s-shutdown-watcher -b -1 --no-pager
journalctl -t k8s-drain -b -1 --no-pager
journalctl -u k8s-node-drain.service -b -1 --no-pager

The hook installation

We start with a one-off cluster setup on your PC or some jump host:

rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: node-drain-agent
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-drain-agent
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "delete"]
  - apiGroups: [""]
    resources: ["pods/eviction"]
    verbs: ["create"]
  - apiGroups: ["apps"]
    resources: ["daemonsets", "replicasets", "statefulsets"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-drain-agent
subjects:
  - kind: ServiceAccount
    name: node-drain-agent
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: node-drain-agent
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f rbac.yaml

# Long-lived token (1 year here; adjust as you like). Requires k8s 1.24+.
TOKEN=$(kubectl -n kube-system create token node-drain-agent --duration=8760h)

# Cluster CA + API server URL, taken from your existing kubeconfig/cluster.
CA_DATA=$(kubectl config view --raw --minify --flatten \
  -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')
SERVER=$(kubectl config view --raw --minify --flatten \
  -o jsonpath='{.clusters[0].cluster.server}')

cat > drain-kubeconfig <<EOF
apiVersion: v1
kind: Config
clusters:
- name: default
  cluster:
    certificate-authority-data: ${CA_DATA}
    server: ${SERVER}
contexts:
- name: default
  context:
    cluster: default
    user: node-drain-agent
current-context: default
users:
- name: node-drain-agent
  user:
    token: ${TOKEN}
EOF

And copy the resulting drain-kubeconfig to /etc/kubernetes/drain-kubeconfig on every node (mode 0600, owned by root). Since the token has a fixed expiry, note a reminder to rotate it (re-run the create token step and redistribute) before it lapses — there’s no controller auto-renewing it.

The files we should setup:

• rbac.yaml — a ServiceAccount + ClusterRole scoped to just what cordon/taint/drain/uncordon needs (not cluster-admin).
• k8s-drain.sh — cordons, taints (homelab.io/shutting-down=true:NoSchedule), and drains the node. Called by the shutdown-watcher below.
• k8s-shutdown-watcher.sh — holds a systemd shutdown inhibitor lock and waits for the PrepareForShutdown D-Bus signal; when it fires, runs k8s-drain.sh before releasing the lock, so the drain genuinely finishes before the OS starts tearing down containers.
• k8s-uncordon.sh — waits for the API server, removes the taint, uncordons. Called on boot by k8s-node-uncordon.service.
• k8s-node-drain.service — wraps the watcher in systemd-inhibit.
• k8s-node-uncordon.service — runs the uncordon script on boot.
• zz-k8s-drain-logind.conf — raises InhibitDelayMaxSec so logind actually waits long enough for the drain to finish.

k8s-drain.sh

#!/usr/bin/env bash
#
# Cordon + taint + drain this node. Meant to be called from
# k8s-node-drain.service's ExecStop as the system is going down.
set -euo pipefail

NODE_NAME="${NODE_NAME:-$(hostname)}"
export KUBECONFIG="${KUBECONFIG:-/etc/kubernetes/drain-kubeconfig}"
DRAIN_TIMEOUT="${DRAIN_TIMEOUT:-120s}"
TAINT_KEY="${TAINT_KEY:-homelab.io/shutting-down}"
TAINT="${TAINT_KEY}=true:NoSchedule"
LOG_TAG="k8s-drain"

log() { logger -t "$LOG_TAG" "$1"; echo "[$LOG_TAG] $1"; }

if ! command -v kubectl >/dev/null 2>&1; then
  log "kubectl not found on PATH, skipping drain"
  exit 0
fi

if [ ! -r "$KUBECONFIG" ]; then
  log "kubeconfig ${KUBECONFIG} not readable, skipping drain"
  exit 0
fi

# If the API server is already unreachable (e.g. this IS the last node,
# or network is already down), don't hang the shutdown forever.
if ! timeout 10s kubectl get node "${NODE_NAME}" >/dev/null 2>&1; then
  log "API server unreachable, skipping drain"
  exit 0
fi

log "Tainting ${NODE_NAME} (${TAINT}) before shutdown"
kubectl taint nodes "${NODE_NAME}" "${TAINT}" --overwrite || \
  log "taint apply failed, continuing anyway"

log "Draining ${NODE_NAME} (timeout ${DRAIN_TIMEOUT})"
if kubectl drain "${NODE_NAME}" \
    --ignore-daemonsets \
    --delete-emptydir-data \
    --force \
    --grace-period=30 \
    --timeout="${DRAIN_TIMEOUT}"; then
  log "Drain completed successfully"
else
  log "Drain failed or timed out — continuing shutdown anyway"
fi

k8s-shutdown-watcher.sh

#!/usr/bin/env bash
set -uo pipefail

LOG_TAG="k8s-shutdown-watcher"
log() { logger -t "$LOG_TAG" "$1"; echo "[$LOG_TAG] $1"; }

log "Holding shutdown inhibitor, watching for PrepareForShutdown"

busctl monitor org.freedesktop.login1 2>/dev/null | \
while read -r line; do
  if [[ "$line" == *"Member=PrepareForShutdown"* ]]; then
    while read -r payload_line; do
      if [[ "$payload_line" == *"BOOLEAN true;"* ]]; then
        log "Shutdown requested, running drain before releasing inhibitor"
        /opt/bin/k8s-drain.sh
        log "Drain step done, releasing inhibitor"
        exit 0
      elif [[ "$payload_line" == *"BOOLEAN false;"* ]]; then
        break
      fi
    done
  fi
done

k8s-uncordon.sh

#!/usr/bin/env bash
#
# Remove the shutdown taint and uncordon this node after boot.
set -euo pipefail

NODE_NAME="${NODE_NAME:-$(hostname)}"
export KUBECONFIG="${KUBECONFIG:-/etc/kubernetes/drain-kubeconfig}"
TAINT_KEY="${TAINT_KEY:-homelab.io/shutting-down}"
WAIT_RETRIES="${WAIT_RETRIES:-24}"   # 24 * 5s = 2 minutes
LOG_TAG="k8s-uncordon"

log() { logger -t "$LOG_TAG" "$1"; echo "[$LOG_TAG] $1"; }

if ! command -v kubectl >/dev/null 2>&1; then
  log "kubectl not found on PATH, skipping uncordon"
  exit 0
fi

if [ ! -r "$KUBECONFIG" ]; then
  log "kubeconfig ${KUBECONFIG} not readable, skipping uncordon"
  exit 0
fi

# Wait for the API server (and this node's kubelet registration) to be reachable.
i=0
until kubectl get node "${NODE_NAME}" >/dev/null 2>&1; do
  i=$((i + 1))
  if [ "$i" -ge "$WAIT_RETRIES" ]; then
    log "API server / node still unreachable after $((WAIT_RETRIES * 5))s, giving up"
    exit 0
  fi
  sleep 5
done

log "Removing taint ${TAINT_KEY} from ${NODE_NAME} (if present)"
kubectl taint nodes "${NODE_NAME}" "${TAINT_KEY}:NoSchedule-" || \
  log "taint removal reported an issue (may simply not have existed)"

log "Uncordoning ${NODE_NAME}"
kubectl uncordon "${NODE_NAME}" || log "uncordon failed"

k8s-node-drain.service

[Unit]
Description=Hold shutdown inhibitor and drain Kubernetes node before shutdown/reboot
Documentation=https://www.freedesktop.org/software/systemd/man/latest/systemd-inhibit.html
After=network-online.target kubelet.service
Wants=network-online.target

[Service]
Type=simple
# systemd-inhibit takes the "delay" lock for the whole life of the wrapped
# process. As long as this is running, logind PAUSES the actual shutdown
# (before any container scopes start getting torn down) until either the
# watcher exits (drain done) or InhibitDelayMaxSec elapses.
ExecStart=/usr/bin/systemd-inhibit --what=shutdown --mode=delay \
  --who=k8s-drain --why="Drain node before shutdown" \
  /opt/bin/k8s-shutdown-watcher.sh
Restart=no
Environment=KUBECONFIG=/etc/kubernetes/drain-kubeconfig

[Install]
WantedBy=multi-user.target

k8s-node-uncordon.service

[Unit]
Description=Remove shutdown taint and uncordon Kubernetes node after boot
After=network-online.target kubelet.service
Wants=network-online.target
Requires=kubelet.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/k8s-uncordon.sh
Environment=KUBECONFIG=/etc/kubernetes/drain-kubeconfig

[Install]
WantedBy=multi-user.target

zz-k8s-drain-logind.conf

InhibitDelayMaxSec=200