AI
Why a Flat Memory Metric Tells You Nothing
Kubernetes pods run fine, memory creeps up, Linux kills the process, the pod restarts, and the cycle repeats. The metric showing that climb does not tell you whether the application has a leak, whether the working set legitimately needs two gigabytes, or whether something else is wrong. CPU profiling reached maturity over decades, from gprof in the 1980s requiring a recompile with -pg, through perf in the 2000s, to today’s always-on continuous profilers. Memory profiling never completed that same arc.
Four Techniques and What Each Actually Measures
Allocation sampling fires on a size threshold, typically every 512 kilobytes allocated, captures the call stack, and produces a time-series you can graph. In-use profiling pairs with that sampler and tracks the net of allocations minus deallocations, making leaks visible. Full heap dumps stop the process entirely to snapshot all memory at one point in time, high overhead and rarely worth it in production. Object introspection, the newest of the four, reads the runtime size distribution of container types to inform smarter data structure choices.
How pprof Connects the Pieces
pprof is both a Protobuf-based data format and a visualization tool. It supports CPU and memory profiles and is close to becoming a standard, with OpenTelemetry’s profiling format integrating alongside it. The instrumentation pprof uses is size-weighted sampling, the same allocation and in-use approach covered earlier. Collecting from one tool and visualizing in another works because the format is shared. Go has pprof support built in. Rust gains it by swapping the memory allocator. Python and Node.js have integrations that expose the same HTTP endpoint.
Parca and Pyroscope: Two Integration Models
Parca scrapes pprof endpoints over HTTP and renders flame graphs and time-series views. Pyroscope, built by Grafana Labs, works through a push model: install the SDK, declare which profile types you want (allocated objects, allocated space, in-use objects, in-use space), and the SDK ships data to the backend automatically. Both deliver allocation sampling and in-use profiling at low overhead. Pyroscope’s flame graphs show encoding JSON and logrus logger as the widest towers, exactly what you would expect given that JSON encoding and string templating allocate heavily.
Object Introspection: Shifting 91% of Allocations Off the Heap
C++ std::vector stores everything on the heap by default. If you know your production container size distribution peaks around 20 elements, switching to Facebook’s Folly small_vector puts 91% of those allocations on the stack and only 9% on the heap. Meta open sourced an experimental project to automate measuring that distribution. Pixie is prototyping the same capability using its dynamic instrumentation layer. A PxL script queries runtime container size distributions so engineers can pick the right inline size before deploying, rather than guessing.
Memory Profiling in the AI Architecture Era
Cloud applications today mostly use host CPU memory. AI workloads change that. Model weights and key-value caches already exceed typical heap sizes. Remote direct memory access means one process writes directly into another host’s memory. GPU memory, accelerators with SRAM-based architectures, and protocols like CXL make tiered memory the norm rather than the exception. The tooling described in this talk covers host memory. Measuring device memory across those tiers is the next unsolved problem, and the need grows with every new accelerator deployed.
Notable Quotes
you don’t really know just from seeing memory use. Is this a memory leak? You know, is the application requesting a sane amount of memory? Dom Delnano · ▶ 04:34
if we used a small vector which is from Facebook’s folly project, um you essentially can get 91% of that data allocated on the stack and only 9% of it will be heap allocated which for certain use cases can really speed things up. Dom Delnano · ▶ 14:20
for memory profiling. You want to prefer sampling based memory tools otherwise they won’t be fit for production. Dom Delnano · ▶ 25:07
it’s still extremely fragmented due to memory specific requirements. Dom Delnano · ▶ 25:38
Key Takeaways
- Flat memory metrics cannot distinguish leaks from legitimate large working sets.
- Sampling-based tools using pprof’s size-weighted approach add low enough overhead for production use.
- Object introspection can move 91% of container allocations from the heap to the stack.
About the Speaker(s)
Dom Delnano is a core maintainer of the Pixie open source project and founder/CEO at Cosmic. He previously worked at CrowdStrike on the eBPF Linux sensor and at New Relic on Pixie full-time. He first built observability tooling at Twitter, and has spent six years applying eBPF across security and observability products.