AI
Why Synchronous Pipelines Break at Scale
Delivery Hero’s food-image pipeline started as a standard Kubeflow prototype. It upscales blurry photos, centers dishes, and adds context around them. The moment the team scaled it to 100 parallel runs, 50 to 80 pipelines sat blocked. Each pipeline carried its own retry logic and thread pools. When the external AI API returned rate limits, every pipeline entered exponential back-off simultaneously. Managing those three separate deployments consumed 60 to 80 percent of engineering time, with almost none left for the actual image work.
The Actor Model as a Fix for Blocked Pipelines
Flattening the pipeline into independent async actors solved the blocking problem. Each step runs as a separate microservice. It reads from a queue, does its work, and writes to the next queue. No central orchestrator decides routing. The routing plan travels inside the message itself. When one actor fails, its error stays local. Other actors keep processing. An LLM-as-judge step can read the generated image, then reroute the message to upload or to retry, without touching any other part of the system.
Asya Architecture: One CRD, Python Handlers, Go Sidecar
Asya defines a single Kubernetes CRD called AsyncActor. A Go sidecar handles all queue communication. The Python container runs the user-written handler, injected via ConfigMap and called over HTTP on a Unix socket. An optional state-proxy sidecar intercepts normal file writes and translates them into calls to external databases or blob storage. An HTTP gateway sits at the edge, accepts standard requests, converts them into queue messages, and returns status back to the caller. The gateway supports both A2A agent-to-agent and MCP protocols.
Separating Business Logic from Infrastructure with Asya SDK
Managing routing across many actors by hand gets complicated quickly. The Asya SDK lets engineers write a single Python flow function describing control flow, including while loops and exit conditions. A compiler transpiles that function into a flat graph. Blue nodes are user-written actors. Orange nodes are auto-generated routers. The compiler also reads existing decorators like retry, maps them to the matching CRD fields, removes the decorator from the handler, and writes the configuration into generated Kustomize base files. User patches layer on top without being overwritten.
Live Demo: Compile, Deploy, Trace
The demo starts from an example repository with actors, a flow, a Dockerfile, and a scaffold YAML. Running compile produces the flat actor graph and CRD manifests. After apply, the asya status command shows all actors in sync with minimum replicas ready. A test request hits the HTTP gateway, passes through actors that retry on failure, and returns a result. Asya ships OpenTelemetry integration out of the box. Prometheus, Tempo, and Loki receive traces automatically, showing the full service graph and per-actor metrics for each request.
Key Takeaways
- 50 to 80 of 100 parallel pipelines blocked until async actors replaced synchronous retry logic.
- Routing state embedded in each message removes the need for a central orchestrator.
- One CRD, a Go sidecar, and pure Python handlers keep abstractions to a minimum.
About the Speaker(s)
Artem Yushkovsky is a Senior AI/ML Engineer at Delivery Hero. He started as a security analyst using static code analysis to find vulnerabilities, then moved into ML engineering for platforms and production use cases. For the past two years he has built generative AI multicloud solutions at Delivery Hero using Kubernetes-native open-source tooling.
Notable Quotes
what we saw in the practice was that you try to run 100 pipelines in parallel and then you see constantly that 50 to 80 are are just blocked Artem Yushkovsky · ▶ 03:47
the information about the routing is embedded inside the message Artem Yushkovsky · ▶ 07:10
in our experience, it was taking 60 to 80 um% of the engineering time just to make it work to achieve this uh productivity on the batch Artem Yushkovsky · ▶ 05:53
long live the queue Artem Yushkovsky · ▶ 24:31