SLOs as the Starting Point for Real Investigations

▶ Watch (1:26)

PostFinance set an SLO requiring fewer than 0.1% of Kubernetes API server requests to fail. Before that target existed, complaints were subjective: “K9s felt slow,” a kubectl get took two minutes. With no objective threshold, the team ignored those signals. Once the SLO dashboard turned red during cluster upgrades, they had a precise timeline to correlate maintenance windows with latency spikes on the read-only API path. That correlation is what drove every fix that followed.

HTTP/2 Persistent Connections and the Unbalanced API Server

▶ Watch (6:01)

Switching to a stacked etcd topology helped but did not stop SLO breaches during upgrades. Grafana showed one API server carrying four to five times the CPU and memory of its peers, with 2 GB of cache versus 200 MB on the idle ones. The cause: HTTP/2 clients open one TCP connection and reuse it indefinitely. All kubelets connected to whichever API server came up first after a restart, ignoring the load balancer entirely. Setting the --goaway-chance flag on the API servers forced periodic connection resets. After a few hours, load and caches equalized across all three servers, and subsequent upgrades produced no user-visible disruption.

Kubernos: Per-Node Health Checks Without Cardinality Explosion

▶ Watch (9:13)

Kubernos is a PostFinance open-source DaemonSet that runs layered health checks on every node: API server by IP, API server by DNS, CNI via the Kubernetes service, and a full ingress-to-pod round trip. It also measures latency between neighboring nodes to catch slow hypervisors. The original design queried every node from every node, producing n-squared requests. At 1,000 nodes that is 1 million checks every 5 seconds. A hash-ring redesign now limits each node to five deterministic neighbors, keeping metrics stable across restarts and avoiding Prometheus cardinality problems.

The 502 Mystery: Mismatched Keepalive Timeouts

▶ Watch (18:17)

A Tomcat-backed ingress application generated 8 to 10 errors per day against 2 million daily requests. The 502s were uniformly distributed across all nginx-ingress pods, appeared in no pattern by endpoint or client, and could not be reproduced with k6 under load. The breakthrough came from reading raw nginx access logs: errors drop the ingress name label and only log the FQDN, costing the team hours. The root error was “upstream prematurely closed the connection.” Tomcat’s default keepalive timeout is 20 seconds. nginx-ingress expects backend connections to stay open for 60 seconds. Requests arriving just as Tomcat sent a FIN packet hit this race condition. The fix: raise Tomcat’s keepalive timeout above nginx’s 60-second value.

Continuous End-to-End Tests Inside the Cluster

▶ Watch (16:04)

PostFinance exposes namespace self-service provisioning, PVC ordering, and automated network-policy creation to application teams. Any of those can break without warning. Rather than wait for user bug reports, the team runs Go-based end-to-end tests as a CronJob every 15 minutes. Tests cover pod creation, PVC mounting, and data writes, pushing results to Victoria Metrics via OTLP. A Grafana dashboard shows pass or fail per cluster per environment. One recent failure showed namespace creation blocked for new users, caught before any developer filed a ticket.

Notable Quotes

it is a as I said quite big bank in Switzerland where we operate on prem kubernetes clusters vanilla kubernetes clusters Clément Nussbaumer · ▶ 00:36

if you do not have really a a target you just ignore the the bad uh services that you could get Clément Nussbaumer · ▶ 01:56

one of the API servers was like had four to five times more CPU and memory usage. Clément Nussbaumer · ▶ 06:17

your observability team will come at you Clément Nussbaumer · ▶ 13:58

we don’t want our users to be the end to end test Clément Nussbaumer · ▶ 16:24

Key Takeaways

  • SLOs turned vague complaints about slowness into timestamped, actionable incidents tied to specific upgrades.
  • The --goaway-chance API server flag forces HTTP/2 reconnections and balances load across all control-plane replicas.
  • Keepalive timeouts between nginx-ingress (60 s) and Tomcat (20 s default) cause rare but real 502 race conditions.

About the Speaker(s)

Clément Nussbaumer is a Systems Engineer at PostFinance, a large Swiss bank running on-prem vanilla Kubernetes clusters. He lives on a farm in Switzerland, applies cloud-native tooling to farm operations in his spare time, and plays music in the evenings.