AI
Why Manual Ops Break Down at Banking Scale
Millennium bcp runs under the EU Digital Operational Resiliency Act. Operational resiliency sits as a board-level topic. That regulatory weight means every automated change needs auditability, controlled change windows, and predictable remediation. The team’s answer to growing platform complexity is not more engineers triaging alerts. The signal volume has outpaced manual review. Too many events, too many P1 and P2 tickets, and engineers stuck cleaning queues instead of building. The control plane approach shifts that burden off people.
Crossplane v2 Watch Operations: The AI Injection Point
Yury walked through the setup: a control plane already managing Azure caches built for the bank, with watch operations running alongside the standard XR compositions. Each watch operation holds a pipeline that calls a function-openai step backed by GPT-4 mini deployed in Azure AI. At roughly five dollars a day during active experiments, the cost is low. Critically, AI fires only on resources that carry an explicit opt-in label. Nothing runs on unlabeled resources. The system prompt tells the model it is a Crossplane cache diagnostician.
Demo 1: Diagnosing a Stuck Cache Without Internal Knowledge
One cache was stuck. Before AI, an operator would drill into XR composition logic, read the event stream, and debug manually. With the watch operation running, the diagnosis appears as an annotation on the cache object. The annotation names the bug and specifies the patch. Yury fixed the cache by activating a second watch operation filtered on two labels: AI enabled and auto-remediation true. The operation patched the parameter ID to the correct value. Yury noted he fixed it without knowing the semantics of the spec.
Demo 2: Cost Analysis and Automated Downscaling
A second cache ran on an XL SKU. A cost-analysis watch operation, enabled by a separate label, compared the SKU against hardcoded cost data in the prompt and wrote a recommendation annotation: downscale from XL to L. A third watch operation, the cost optimizer, reacts when a third label, allow-cost-optimization true, is set. Setting that label triggered the downscale automatically. The spec confirmed the change: parameters moved from XL to L. Cost data can come from an external database in production; the demo used an inline prompt for clarity.
Keeping AI Safe in a Regulated Environment
Every action is schema-bound. AI reasoning produces a bounded proposal and nothing changes outside the targeted operation. Change windows and policy constraints are enforceable at the label level. Nuno argued that prompts should come from observing what engineers already do manually, then encoding that practice. Data sovereignty matters too: local models are on the roadmap for cases where sending data to a hosted API is not acceptable. The team also pointed to evidence trails. An LLM writing structured content into an incident tracker beats an engineer typing notes at 2 a.m.
What Comes Next at Millennium bcp
The cache demo covers one resource type. Millennium bcp supports a few dozen resource types in Crossplane, and the plan is to enable AI-assisted analysis across all of them, starting with databases and messaging. Crossplane v2 app-level claims mean an annotation could surface directly on an application claim telling a developer exactly what is broken. Reporting to the European Central Bank on incidents also improves because structured machine-written data replaces free-text boxes. Adoption rolls out namespace by namespace, matching the automation level to workload criticality.
Q&A
How do you prevent label tampering in a multi-tenant cluster from breaking or hijacking operations? Yury noted that an SRE overriding an AI annotation with a better value (moving to medium instead of large) simply improves the outcome, and removing an annotation to exclude a resource from AI is a deliberate choice someone should own. ▶ 27:56
Notable Quotes
it was around five bucks a day. So, uh, and it will be cheaper in a standard operation, right? Yury Tsarev · ▶ 09:11
we are fully under controlling what is happening within the control plane and we invoking AI only intentionally. Yury Tsarev · ▶ 10:02
at 2 a.m. when when your system is down and you need to know bring it back up Nuno Guedes · ▶ 22:14
No one’s going to forget to get the right data into the incident tracking system if it’s in the operation. Nuno Guedes · ▶ 23:48
AI becomes safe in production for instance in places like ours where we get a governed control plane doing it. Nuno Guedes · ▶ 26:41
Key Takeaways
- GPT-4 mini watch operations cost around five dollars a day during active experiments.
- Labels gate AI access per resource, keeping every action schema-bound and auditable.
- Start with read-only diagnosis annotations before enabling auto-remediation labels.
About the Speaker(s)
Nuno Guedes is Head of Public Cloud at Millennium bcp, Portugal’s largest privately-owned bank, and a Microsoft MVP for Azure. He leads cloud-native transformations focused on containers and serverless, specializing in solving complex challenges through teamwork.
Yury Tsarev is a Principal Solutions Architect at Upbound and the creator of k8gb. He is an active contributor to the Crossplane ecosystem and speaks regularly on control planes and distributed systems.