AI
KServe: Easy Serving, No Built-in Security
KServe started as a KubeFlow sub-project and now handles inference workloads for any runtime, including vLLM, SGLang, and traditional ML models, through a single custom resource. One CR and the model serves. KServe also ships a new LLMInferenceService CR built for LMMD integrations, covering KV cache, gateway wiring, and disaggregated inference with separate prefill and decode nodes. What it does not provide is security. That responsibility sits entirely with the team deploying it.
Why AI Workloads Need Zero-Trust Treatment
KServe is designed to be easy, not secure. Before serving anything, teams must verify the model or agent matches what was approved, validate signatures and attestations before loading weights, and trace back to the exact training run and dataset if hallucinations appear in production. Regulated environments add a harder requirement: provable chain of custody kept intact from training through deployment. Most open-source solutions do not provide this out of the box. The gap between what KServe gives you and what a production security posture requires is where KitOps fits.
KitOps: OCI Packaging for the Full AI Artifact Set
KitOps is a CNCF sandbox project that packages model weights, datasets, prompts, and config into a single OCI artifact stored in any standard registry. Every layer gets a SHA-256 digest. Content-addressable storage matters when payloads run into hundreds of gigabytes: deduplication across model versions cuts transfer and storage costs that a flat S3 bucket cannot match. A $30 billion European logistics company operating 3,000 offices across 90 countries standardized on KitOps specifically because it gives a single, unified chain of custody across Edge, on-prem, Azure, Google, and AWS deployments simultaneously.
Three Security Levels: From SHA Checks to Registry Quarantine
Level one is the minimum: a kit pull checks SHA digests and a cosign verify confirms the signature before KServe loads the model. A failed check sends the pod into crash loop backoff and it never serves. Level two adds admission policies at the cluster, blocking models that have not passed specific security checks or performance benchmarks before reaching production. Level three moves the gate earlier: a pull policy applied at the registry itself prevents anyone, including developers on local laptops, from pulling an artifact until it clears a quarantine scan. This stops MCP servers and agents, which have documented exploits for privilege escalation and data exfiltration, from spreading across the organization before review.
Live Demo: Attested Model Serves, Unattested Model Stays Blocked
Gavrish Prabhu packed a GPT-OSS 20B model from Hugging Face into KitOps, pushed two versions to the Jozu registry (one attested at 12.8 GB, one unreviewed), and deployed both as KServe InferenceServices. The attested model passed its init container verification, downloaded via a custom storage container, and ran vLLM 17.1 until it accepted OpenAI Python library chat completion requests. The unattested model printed “blocked” in its init container logs and entered crash loop backoff. GPU resources were never allocated to it. Both deployments used the same ISVC definition with only the model tag changed.
Q&A
Should MLOps teams with hundreds of data scientists adopt attestations even without a compliance mandate? Micklea recommended yes: wrap KitOps in a notebook or MLflow template so scientists fill in the middle and the SDK handles packing and pushing automatically, then use attestations to keep all approval and evaluation criteria with the artifact so any tool or team can read them without a vendor connection. ▶ 27:38
Can KitOps reference terabyte-scale datasets stored remotely rather than bundling them? S3 reference support was recently merged and will land in the current or next release; in the meantime, adding a reference file inside the model kit preserves the audit link but cannot guarantee the dataset is byte-for-byte identical to what was used at training time. ▶ 31:55
How does KitOps work alongside platforms like Weights and Biases? Platforms typically store artifacts on a filesystem, so a pipeline can call the kit CLI on any file system change to pack and push; a native Weights and Biases plugin does not exist yet, but an MLflow storage backend is in progress. ▶ 33:49
Notable Quotes
it is not designed to be an exceptionally secure. It’s meant to be exceptionally easy. Brad Micklea · ▶ 02:31
allowing people to just choose random MCP servers and agents off the net and pull them onto the local machine, not really the safest thing you can do, to put it mildly. Brad Micklea · ▶ 16:40
So it says that this one is not attested by your public key and it’s not ready to go. So it’s saying blocked. Gavrish Prabhu · ▶ 23:48
We see KOPS as the gate between development and production. Securing that gate is critical and this is a great way to do it. Brad Micklea · ▶ 25:34
Key Takeaways
- KServe makes inference easy but ships no security; teams must add it themselves.
- KitOps SHA-256 digests and cosign attestations block unreviewed models before GPU allocation.
- Registry-level quarantine policies stop unscanned agents and MCP servers from reaching any machine in the organization.
- A $30 billion logistics company with 90-country operations standardized on KitOps for unified chain of custody.
- KitOps reached nearly 250,000 downloads as a CNCF sandbox project with the S3 dataset reference feature recently merged.
About the Speaker(s)
Brad Micklea is Founder and CEO of Jozu and a maintainer of the KitOps project and Model Pack specification inside CNCF. He previously ran Amazon API Gateway as GM. Before that, his startup Codenvy was acquired by Red Hat, where he became VP of Developer Tools and Marketing.
Gavrish Prabhu is a Founding ML Engineer on the Nutanix Enterprise AI team with a background in distributed systems. He maintains both KServe and the Envoy AI Gateway project, with a focus on next-generation AI inference systems.