AI
The Platform at Scale: 1,000 Control Planes and One Unified Interface
Before API contracts, Allianz Technology’s 15 product teams each had their own interface. One team required a GitHub issue. Another wanted a Jira ticket. A third expected an email. Four years after starting the Crossplane-based platform, that fragmentation is gone. Today 1,000-plus customer control planes share a single interface: teams manage their infrastructure via Kubernetes claims stored in Git, synced to their control plane by Argo CD. The platform now tracks 20,000-plus infrastructure claims across 90-plus products.
How Product Teams Build and Ship on the Platform
Each product team defines an XRD schema, writes a Crossplane composition (Python is an option), adds tests, then opens a pull request. CI builds and tests automatically. Releases go out as canary deployments first. With 1,000-plus customers, shipping to all of them at once risks taking down everything. A canary to five customers catches real-world failures before full rollout. Once the canary holds, the PR merges and CI publishes the package to all control planes.
API Contracts: Replacing Guesswork with Typed Promises
Unclear ownership caused real pain. Super-compositions forced teams into tight coupling. Nobody knew exactly what resources a landing zone claim would produce: a VPC, subnets, something else entirely? The API contract answer is an XRD with three parts. First, typed input fields with enum and string validation. Second, documented status fields, for example a subscriptionId that downstream teams can rely on. Third, explicit dependency declarations so the database team knows it depends on the subscription team’s output. Versioning is hard in Kubernetes; avoid breaking changes where possible.
Cross-Provider Integration Without Super-Compositions
Same-provider field references are straightforward. Provider Azure can resolve selectors within its own resources. The hard case is provider SQL consuming a subscription ID owned by provider Azure. Provider SQL has no knowledge of provider Azure’s types. The solution is Crossplane’s required-resources feature (called extra-resources in v1). It imports any composite resource or managed resource from the control plane API. The database team reads status.endpoint and status.port from the SQL instance XR and passes only a sqlInstanceRef in the app database claim. No KCL code from the infra team is exposed to the app team.
Composition Tests: 95% Coverage Before End-to-End Runs
Strong API contracts on status fields make offline testing practical. Using crossplane render, the desired resource stream is produced locally and assertions run against it. Yury described writing TDD-style tests that let him progress at a dramatically faster pace, leaving chainsaw-style end-to-end tests only for the final pipeline stage. Open-source frameworks like xp or Cisco BDD work with the same approach. The goal: cover 95% of composition logic locally, keep end-to-end tests short, and only promote to real infrastructure after local assertions pass.
Lessons Learned: Breaking Changes, Drift, and Zero Production Incidents
One provider change broke 50-plus teams and was discovered only in production. Rolling it back was painful. Contract testing plus versioning stopped that class of failure. For versioning, Allianz avoids conversion webhooks when possible. They migrate customers explicitly from a v1 XRD kind to a v2 XRD kind. Documentation drift is handled by writing docs as annotations and autogenerating output. Examples are tested alongside the composition, so they stay accurate. The result is near-zero production incidents and high adoption across internal customers.
Q&A
How do you handle resources shared across multiple clusters? The team does not share state across control planes today, but suggested provider Kubernetes for syncing or management-policy observe to read cloud-provider API data across clusters. ▶ 27:48
Any tips for avoiding painful XRD version migrations? Design XRDs for evolution from the start: use enums with a single value rather than bare strings so the field can grow without breaking. Also run kube-api-linter against XRDs during authoring to catch non-scalable patterns early. ▶ 29:00
Have you migrated to Crossplane v2 yet? Migration is in progress. v2 providers are already in use. The main architecture will not change much; the primary v2 benefit would be namespaced resources, which Allianz does not need because each customer already has a dedicated control plane. ▶ 30:08
Notable Quotes
don’t under underestimate the effort you should put into defining this API Florian Hopfensperger · ▶ 05:16
the connection secret is kind of weak contract because whenever the field in the connection secret that is not usually visible it’s not on the top it’s going is going to be renamed by intro team everything will break for uh for the consumer Yury Tsarev · ▶ 17:43
provider changes something and broke 50 plus teams. So discovered only production and the roll back was a nightmare. Florian Hopfensperger · ▶ 25:26
Documentation is already outdated when it’s written. This is my takeaway always Florian Hopfensperger · ▶ 25:29
with a uh with this uh local tests you can cover the 95% of the uh of your composition logic Yury Tsarev · ▶ 23:26
Key Takeaways
- API contracts on XRD status fields let provider SQL consume provider Azure data without shared code.
- Canary deployments to five customers before full rollout protect 1,000-plus control planes from bad releases.
- Local composition tests with crossplane render cover 95% of logic before any end-to-end pipeline run.
About the Speakers
[Florian Hopfensperger] is a Software Engineering Lead at Allianz Technology working on Kubernetes Control Planes. He contributes actively to Cloud Computing, GitOps, and open source software.
[Yury Tsarev] is a Principal Solutions Architect at Upbound. He created k8gb and contributes actively to the Crossplane ecosystem. He speaks regularly on control planes and distributed systems.