AI
Why GPUs Cannot Share Time Like CPUs
A CPU slices time across processes, saving and restoring state in microseconds. A GPU does not. When a kernel starts running, it monopolizes the entire GPU until it finishes. Nothing else runs in parallel. A long-running kernel blocks everything waiting behind it. That behavior is deliberate: GPUs skip the prediction hardware CPUs spend transistors on, spending everything instead on raw computation. The trade-off works for repetitive matrix math. It creates real problems when you try to share a GPU across workloads.
GPU Memory: Chunks, Not Exact Allocations
Ask a GPU for 8 KB of memory and it gives you a 2 MB pool. A second 8 KB allocation draws from that same pool. nvidia-smi shows 2 MB used regardless. If you need more, the driver grabs an even bigger chunk. Two programs sharing a GPU can each see all 16 GB of combined allocation, not just their own portion. The driver holds extra memory that your process never learns about. Speed and simplicity drive this design, but it means you cannot know exactly how much memory your process is actually holding.
CUDA Context: How a Program Claims a GPU
Every program that wants GPU access opens a CUDA context. That context tracks copied memory, queued kernels, GPU state, and scheduling. Data lives in RAM first. Calling cp.array copies it into GPU memory. The computation runs there, produces a result, and then the result copies back to RAM before the CPU can use it. Multiple programs each open their own CUDA context. Each one goes through that full copy-compute-copy cycle. The GPU driver, not the Linux kernel, owns all of this. The source code is not public.
Four Layers Between a Pod Spec and a Working GPU
Kubernetes has no container GPU interface. To bridge that gap, four components stack on top of each other. The Nvidia driver installs into the Linux kernel and exposes each GPU as a file descriptor. The Nvidia container toolkit hooks into the container runtime (containerd, for example), relaxes namespace constraints, injects the driver, and sets environment variables before the container starts. The Nvidia device plugin runs as a DaemonSet, registers each GPU with the kubelet, and keeps Kubernetes informed of GPU availability on each node.
End-to-End: From kubectl Apply to GPU Pod Running
A pod requesting nvidia.com/gpu: 1 hits the API server, clears authentication and authorization, and lands in etcd. The controller manager creates the pod object. The scheduler filters nodes to those advertising a GPU resource, scores them, and binds the pod to the best match. The kubelet on that node calls containerd. The container toolkit hooks fire before the container starts, injecting drivers and environment variables. The device plugin confirms GPU allocation. The pod starts with a full GPU available and correct drivers in place, with none of that visible to the container itself.
Notable Quotes
GPUs do not do that at all. When a kernel starts running on the GPU it takes over the entire GPU until it’s completely done. Gulcan Topcu · ▶ 09:09
Nvidia wrote it, shipped it, we just install it. We call the API, something happens inside, and we get a result back, but what actually happens in between? We don’t know because the source code isn’t public. Gulcan Topcu · ▶ 15:54
this driver is basically the owner of the hardware. Whereas most of the time when you use CPU and RAM, you’re going through the Linux kernel. When you’re using a GPU, you are basically using a binary which we have no control over, which is closed source, we cannot inspect whatsoever Daniele Polencic · ▶ 24:17
Key Takeaways
- A GPU kernel runs to completion with no preemption, blocking every waiting workload.
- Requesting 8 KB of GPU memory allocates a 2 MB pool; exact usage stays hidden.
- Four Nvidia components (driver, toolkit, device plugin, scheduler awareness) are required to run a GPU pod in Kubernetes.
About the Speaker(s)
Gulcan Topcu is a Kubernetes and security engineer who designs secure, scalable cloud-native systems and automates infrastructure with Python and Go. She co-authored a book with the LearnKube team covering real Kubernetes production stories.
Daniele Polencic teaches containers and Kubernetes at LearnKube and holds a Kubernetes Administrator certification from the Linux Foundation. Over the past decade he has trained engineers at companies across e-commerce, finance, and the public sector.