AI
Why the Single-Cluster Architecture Broke Down
After KubeCon 2024, Bloomberg’s Trino platform grew enough that data governance stopped being the hard problem. Availability did. The old architecture exposed zone-specific endpoints, so if a service owner deployed into two zones, they got two separate endpoints back. Disaster recovery existed only if service owners and query users wired it into their own applications. Trino queries are stateful and tied to a single coordinator. When that coordinator fails, the query is lost, not retried.
Karmada as the Multi-Cluster Orchestration Layer
Karmada, a CNCF incubation project with Bloomberg as a maintainer, distributes Trino deployment CRs to member clusters without requiring any changes to the application itself. A propagation policy tells Karmada to duplicate each Trino deployment across all member clusters so every cluster runs Trino pods. Karmada continuously monitors deployment health. If a Trino deployment in a member cluster becomes unhealthy due to node failures or resource pressure, Karmada detects it and attempts to recreate the pods. Bloomberg presented Karmada’s use across the organization at KubeCon China 2024.
Trino Gateway: One Query Endpoint Across All Data Centers
Trino Gateway acts as a load balancer and proxy that routes queries across multiple Trino clusters. It backs configuration in a database and exposes a single federated endpoint. When a Trino service comes up healthy, the Trino service controller creates a registration CR that tells the gateway its address. As Trino pods scale up, scale down, or disappear during a cluster upgrade, the gateway automatically registers and deregisters them. Query users always connect to one endpoint regardless of which data center actually runs their query.
How the System Responds to Failures
Three failure scenarios show the system’s behavior. First, if a gateway pod in one data center fails, service discovery stops routing traffic to it. The surviving gateway in the other data center can still dispatch queries to Trino pods in the failed data center, so compute is not lost. Second, if a Trino pod fails, the gateway detects the unresponsive backend and reroutes queries to healthy pods. On recovery, it rediscovers the pod and resumes distributing load. Third, if the entire API server goes down, the stretched etcd backend retains all state. After nodes and pods restart, the controller resyncs all resources and operation resumes.
Roadmap: HPA Scaling, Query Logging, and Resource Groups
The dual data center setup is the current release. Bloomberg plans to extend to additional data centers for wider fault tolerance and geographic distribution. Worker pod counts are statically defined today via an internal algorithm. The team is moving toward HPA-driven scaling that adjusts worker replicas based on query load, CPU utilization, and JVM metrics so clusters scale up at peak demand and release resources during idle periods. Two more items are planned: capturing query metadata via event listeners, and enforcing resource group concurrency limits to prevent a single heavy query from monopolizing cluster resources.
Q&A
How do you handle rolling updates without downtime across clusters? Blue-green deployment responsibility currently stays with the service owner. Bloomberg plans to use Karmada propagation policies later in the year to deploy to one zone first and validate it before rolling to the other. ▶ 18:32
How do you validate disaster recovery automatically? Trino Gateway has built-in application logic that detects whether a backend is healthy enough to receive routed queries, and Bloomberg runs scheduled and release-based end-to-end tests. ▶ 19:53
What happens if Karmada itself goes down? At Bloomberg, Karmada is a managed service with its own SLOs. The Trino team does not operate it directly. ▶ 21:14
Why Trino Gateway instead of the Kubernetes Gateway API? Trino queries are stateful. The coordinator returns a query ID that clients must use for subsequent pages. Trino Gateway also supports tagging backends as inactive, letting in-flight queries finish on that backend while blocking new queries from routing to it, enabling clean drains. ▶ 23:54
How hard is it to add a third cluster? Adding a cluster at Bloomberg means requesting machines via their internal Kubernetes-as-a-service tooling, deploying the stack with their BKA deployment tool, and registering the new cluster with managed Karmada via API. The workflows for service owners and query users stay identical. ▶ 25:48
Notable Quotes
The core challenge with Trino is that queries are stateful and tied to a specific coordinator. If that coordinator fails, the query doesn’t transparently fail over. It is simply lost. Sung Yun · ▶ 02:03
multicluster resilience becomes an implementation detail that is fully owned by the platform. Sung Yun · ▶ 05:54
The API reconnects to CD. Our controller resync all the resources and the system resumes operation as if nothing happened. Antoine Marthey · ▶ 15:47
So we only lost the gateway but not the compute behind it. Antoine Marthey · ▶ 14:27
Key Takeaways
- Karmada fans out Trino deployment CRs to all member clusters, so every data center runs Trino pods by default.
- Trino Gateway provides a single query endpoint and drains backends gracefully so in-flight queries finish before rerouting.
- A stretched etcd backend shared across API server pods means no state is lost even during a full infrastructure restart.
About the Speaker(s)
Sung Yun leads Bloomberg’s Cloud Native Compute Services Trino and Iceberg Engineering team. His team builds a data exploration platform on Kubernetes, Trino, and Apache Iceberg. Sung is an Apache Iceberg Committer and holds a master’s degree in computer science from Cornell University.
Antoine Marthey is a Software Engineer on the same CNCS Trino and Iceberg team at Bloomberg, focused on building a multi-tenant data exploration platform using Kubernetes and Trino.