AI
The Problem: 10,000 Expensive Machines Working Six Hours a Day
A financial firm runs approximately 10,000 high-cost trading systems. Each machine works for six hours, then sits unused for the rest of the day, still under audit. Any system found out of compliance with auditor expectations gets pulled from the trading pool. The business question was direct: how do you maximize capital expenditure on hardware that is idle 75% of the time without touching the audit posture those systems must maintain?
Hosted Control Planes and the Ephemeral Cluster Architecture
The architecture uses Cluster API’s hosted control plane pattern. The management cluster runs child-cluster control plane components (API server, controller manager, cloud controller) as pods. MetalLB provides a dedicated IP pool for those control planes on-prem. Mirantis’s open-source Cordant project unifies state and cluster management. Kyverno enforces policy across the estate. Flux handles GitOps delivery. When spot hardware is released at the end of a trading day, workers join existing clusters rather than requiring new clusters to be created from scratch.
Transparent Confidential Container Injection
Every deployment arriving at a child cluster gets its runtime class silently switched to the Kata confidential containers runtime via a custom mutating admission webhook. Developers push a deployment; the webhook rewrites the runtime class to kata before the pod schedules. Namespaces like kube-system and Flux are excluded. The team deliberately avoided routing this through Kyverno: if Kyverno failed, nothing would mutate and nothing would run, making the failure mode too broad. Kubernetes 1.35 now makes mutating admission a first-class feature, removing the need for the custom webhook going forward.
Kyverno Policies That Back the Audit Trail
Kyverno handles validation, not mutation. Policies confirm that every deployment carries the Kata runtime class, that all images originate from the internal Artifactory registry, that resource limits are set, and that resource names match what auditors expect to find. The entire environment is air-gapped. All images, K0s binaries, bootstrap providers, and Cluster API providers live in the internal registry. Nothing reaches an external endpoint.
FiveSpot: Time-Based Spot Scheduling for Regulated Hardware
Markets close at 6 p.m. At 7 p.m. the spot pool opens. Teams owning idle hardware submit their machines with a window specifying when the hardware can be borrowed and when it must be returned. FiveSpot, a tool written by engineer Eric Bourgeois, manages graceful shutdown, graceful pause, emergency kill, and time-based scheduling. By 5 a.m. servers drain; by 6 a.m. everything is back to normal. FiveSpot has been donated to the Fintech Open Source Foundation and is working through that governance process now.
Q&A
What workloads run on the overnight clusters? Internal teams use the capacity for experiments, tests, and development. Nothing production runs there, partly because the clusters disappear within hours. ▶ 23:59
Notable Quotes
Security is never a problem. William Rizzo · ▶ 15:33
providing to the developers this transparent injection of runtime, they don’t have to think about it. They don’t even have to necessarily know. William Rizzo · ▶ 16:34
it will die in in matter of hours. William Rizzo · ▶ 24:40
Key Takeaways
- 10,000 trading machines sit idle 18 hours a day; spot scheduling reclaims that compute for internal teams.
- A mutating admission webhook injects Kata runtime class transparently, cutting developer compliance work to zero.
- FiveSpot enforces time-based machine lifecycles and is being open-sourced via the Fintech Open Source Foundation.
About the Speaker(s)
William Rizzo is Global Field CTO at Mirantis, a Kairos maintainer, and a CNCF and Linkerd Ambassador. He works on the Cluster API provider integration inside Kairos and contributes to FINOS, the open-source in finance foundation. His focus spans Edge, AI, and platform engineering.