AI
Ramalama: AI Models as OCI Artifacts
Ramalama wraps AI model management in the same commands container engineers already use: pull, run, push. Models are stored as OCI artifacts and pulled from registries including HuggingFace. A fine-tuned model can be pushed to any container registry, shared across teams, and run with GPU access through container image-based discovery. Ramalama uses Podman under the hood to run the LLM in a container. The dry-run flag prints the full container command, which feeds directly into Podman pod setup.
Podman Pods Isolate the LLM and Agent
Podman runs containers rootless and daemonless, which matters for agentic AI. An agent with filesystem access can delete your home directory. Podman’s pod model puts the LLM container and the agent container together in a shared network namespace. They communicate over localhost inside the pod. Nothing reaches the host. The same OCI image runs locally in Podman and in production. Podman became a CNCF sandbox project in 2024.
Local Demo: Llama 3 and Goose in a Shared Pod
The demo used ramalama pull to fetch Llama 3 from HuggingFace as an OCI artifact, then ramalama dry-run to extract the container command. A Podman pod named with open ports came next. Llama 3 started as the server container. Goose, an agentic AI platform, ran as the second container in the same pod. With both containers sharing the pod network, Goose pointed at localhost 8080 to reach the Llama 3 instance. The agent answered “what is a container in one sentence” without touching the host.
From Podman to Kubernetes with One YAML
ramalama convert turns the model into a container image with the model baked into /models. podman generate cube takes the running pod ID and outputs Kubernetes YAML. That YAML needs one edit: the local volume mount doesn’t exist in the cluster, so an init container using the converted car image copies the model into a shared emptyDir volume. After loading images into minikube with minikube image load, kubectl apply deployed the same Llama 3 and Goose pod as a Kubernetes Deployment. The response took two and a half minutes without a GPU.
Quadlet Runs the Same YAML as a systemd Service
Quadlet, a Podman feature, runs containers as systemd services. It accepts a .kube file pointing to the same Kubernetes YAML used for the minikube deployment. Systemd handles start at boot, automatic restarts, and updates. Port publishing and service configuration go in the Quadlet file. systemctl start and systemctl enable are all it takes. The result: one YAML definition runs the Llama 3 and Goose pod on a local Podman machine, in a Kubernetes cluster, and on an edge device.
Key Takeaways
- Ramalama pulls AI models as OCI artifacts using familiar container commands, with GPU discovery built in.
- Podman pods isolate the LLM and agent from the host, preventing filesystem damage during local development.
podman generate cubeproduces Kubernetes YAML from a running pod, cutting most of the migration work.- Quadlet bridges the gap for edge and single-node deployments using the same Kubernetes YAML without a cluster.
- One YAML definition runs the same workload across local Podman, Kubernetes, and systemd environments.
About the Speaker(s)
Ashley Cui is a Software Engineer at Red Hat working on container tools including Podman.
Urvashi Mohnani is a Principal Software Engineer on the OpenShift Container Tools team at Red Hat. She has contributed to and maintained open source container projects including Podman, Buildah, CRI-O, and Skopeo, and is a co-organizer of DevConf.US.