Project Lightning Talk: Still Burning GPUs On Debugging? Scale AI In One Line - Anna Kramar, Maintainer

by Anna Kramar

Day 2 · Elicium 2 · Project Lightning Talk · 5 min · 3 min read

Containers & Orchestration Cloud Architecture

TL;DR

Anna Kramar demonstrated how the Kubeflow SDK lets data scientists scale a training function from a local subprocess to 800 GPUs by changing one line of configuration. No edits to the training function itself. The SDK reached 140,000 downloads in recent months and already includes clients for training, hyperparameter optimization, model registry, and Spark distributed data processing, with pipelines, feature store, and an MCP server in progress.

The Real Cost of Kubernetes Infrastructure Friction

▶ Watch (0:03)

Data scientists build a training function, run it locally, and then hit Kubernetes. Pods, CRDs, and YAML files take over. Debugging replaces building. Every failed run burns GPU hours, and GPU hours cost real money. Kramar framed this as the core problem Kubeflow SDK exists to solve. The gap between “works on my machine” and “runs at scale on Kubernetes” is where productive AI work stalls.

One Line to Switch from Local to 800 GPUs

▶ Watch (1:31)

The SDK’s trainer_client runs a training function locally using LocalProcessBackendConfig, which executes it as a subprocess. To scale the same function to distributed training, switch to the Kubernetes backend, set the number of nodes, and set resources per node. Kramar’s demo distributed one function across 800 GPUs with a few added lines. The training function itself required zero modifications. Switching between subprocess, Docker, Podman, and Kubernetes changes one line.

What the SDK Ships Today

▶ Watch (3:19)

Four clients are available now. trainer_client handles fine-tuning and distributed training. optimizer_client runs hyperparameter optimization after training. model_registry_client manages model versioning and artifact storage. A spark_client, added recently, handles distributed data processing on Spark. Each follows the same Pythonic API pattern. The goal is one SDK entry point for the full ML workflow, installed with pip install kfp.

What Is Coming Next

▶ Watch (3:58)

A pipelines client for workflow orchestration and a feature store client are already in development. OpenTelemetry integration for observability and MLflow for experiment tracking are planned. Kramar also mentioned an MCP server so AI agents can operate Kubeflow directly. The project hit 140,000 downloads in recent months. A blog post covering the latest release is published, and maintainer sessions are available at the conference this week.

Notable Quotes

every fail costs a lot of money because of the GPU hours you spent Anna Kramar · ▶ 01:03

you can change your environment variable really easily with just one line Anna Kramar · ▶ 02:42

we recently hit um 140,000 downloads for recent months of CFL SDK Anna Kramar · ▶ 04:33

Key Takeaways

Switching from local to 800-GPU distributed training requires changing one backend config line.
Four SDK clients ship today: trainer, optimizer, model registry, and Spark.
The project crossed 140,000 downloads recently, with MCP server support planned.

About the Speaker(s)

Anya Kramar is a Software Engineer at Red Hat on the Kubeflow Developer Experience team. She works on developer workflows and user experience within the Kubeflow and OpenShift AI projects, with a focus on making machine learning more accessible to developers.

Filed under

Containers & Orchestration Cloud Architecture