AI
Why Feedzai Runs Spark at This Scale
Feedzai’s fraud detection platform processed 90 billion events last year, totaling 83 trillion dollars in payments. Spark handles batch feature engineering and model training. Delayed Spark execution degrades transaction scoring and can leave fraud undetected. The platform runs over 150 production clusters isolated per client and use case, launching 4,000 Spark applications each day, which works out to three per minute across several regions. That number keeps growing as Spark-as-a-service enables new integrations.
Building Spark-as-a-Service on Kubernetes
Feedzai migrated from independent YARN clusters, each requiring SRE involvement for basic tasks like reading Spark logs, to a Kubernetes-native model with three requirements: a serverless-feeling API and UI for users, zero platform-team involvement in daily operations, and proper multi-tenant workload isolation. Volcano provides hierarchical queues that replicate the old per-client cluster boundaries inside a single Kubernetes cluster. A single Spark submission creates 15 Kubernetes objects, including submission jobs, drivers, executors, pod groups, config maps, secrets, and roles. A 150 KB JSON payload becomes 700 KB of manifest data across those objects.
How MVCC Turns 150 KB into 4.8 GB
Etcd uses multi-version concurrency control, the same mechanism in PostgreSQL and MySQL. Every status update to a Kubernetes object writes a full copy, not a diff. A Spark application passes through five phases during its lifecycle, producing six full versions. One 150 KB object becomes 1 MB in etcd. In a benchmark submitting 3,000 jobs over 15 minutes, starting from a 270 MB etcd baseline, the load added roughly 700 MB of logical object data but the reported physical etcd size reached 4.8 GB, almost five times the expected figure. Cleanup made it worse: deleting jobs pushed storage from 3.5 GB to 4.8 GB because controllers watching deleted resources fired additional events.
Fragmentation: Why Compaction Is Not Enough
Kubernetes runs etcd compaction every five minutes to remove outdated MVCC revisions. Compaction reclaims logical space but does not return bytes to the host filesystem. Freed chunks scatter across the backing store. New objects write into the next available contiguous chunk, so physical size grows even after logical size shrinks. Defragmentation reorganizes physical space, but on several cloud providers it runs only every two days or after a high fragmentation threshold is crossed. Some providers ship managed Kubernetes clusters with only 250 MB of etcd storage, making fragmentation a hard capacity ceiling, not a background nuisance.
Offloading Spark Objects via the Aggregation Layer
The Kubernetes API server has three layers. Core objects and CRDs both persist to etcd. The aggregation layer proxies requests to a custom server you control, so objects handled there never touch etcd. Kube-metrics-server works exactly this way: pod and node metrics stay in an in-memory store inside the server. Feedzai applied the same pattern to the Spark operator API group, registering an APIService object pointing to a custom server with an in-memory store. The live demo showed 300 Spark applications created against the custom server, confirming objects reached it instead of etcd, then showed they vanish on server restart, which is the known trade-off of in-memory storage.
Benchmark Results and Design Principles
Running the same 3,000-job, 15-minute benchmark with the aggregation layer active, objects created increased 30% and logical size grew 20%. Etcd physical storage dropped 45%, saving 2 GB. All MVCC churn for Spark application objects moved to the custom server, cutting etcd bloat and fragmentation. The key design lessons: keep custom resources under 10 KB, minimize status update frequency to reduce revision churn, and design for deletion using TTL controllers or finalizers with timeouts. Long-term retention belongs in dedicated storage, not etcd.
Notable Quotes
CRDs are not free. Size has a cost. So large objects bigger than 10 kilobytes will multiply over the MVCC revisions that we’ll create. João Azevedo · ▶ 25:52
So this is actually what you get when you rely on an in-memory store. João Azevedo · ▶ 24:10
150 kilobytes of an object quickly became one megabyte of storage on our CD. João Azevedo · ▶ 17:01
deleting jobs uh storage went up instead of going down João Azevedo · ▶ 14:22
Key Takeaways
- A 150 KB Spark job payload generates 700 KB of Kubernetes manifests across 15 objects.
- MVCC versioning multiplied one Spark application object from 150 KB to 1 MB in etcd.
- Delegating Spark objects to a custom aggregation-layer server cut etcd storage by 45%, saving 2 GB.
About the Speaker(s)
João Azevedo is a Senior Data Reliability Engineer at Feedzai with a background in database research. He works on high-performance, data-intensive architectures spanning transactional databases and streaming systems.
João Soares is a Platform Engineer at Feedzai focusing on compute and networking. He also serves as an assistant professor at the University of Coimbra teaching Distributed Systems and Compilers, with research interests in formal architecture verification.