AI
Why Tail Sampling Gets Expensive at Scale
Tail sampling forces every span to travel to a central OpenTelemetry Collector before any decision is made. Each span averages 1 kilobyte. Holding a configurable time window of spans in memory turns the Collector into a bottleneck. When vertical scaling runs out, operators add horizontal Collector instances and a trace-ID-aware load balancer to route spans correctly. In multi-region deployments, sending 100% of spans across region boundaries before dropping 99% of them generates real cloud egress costs.
How Retroactive Sampling Works
Spans carry far more data than a sampling decision needs. If the rule is “keep traces longer than 5 seconds or containing an error,” only start time, end time, and status matter. Retroactive sampling exploits this. A local agent buffers incoming spans in a FIFO queue and extracts only those key fields. It sends that stripped payload to a central sampling server. The server decides, returns a set of trace IDs, and the local agent then reads the queue sequentially, forwarding matching spans and discarding the rest.
Benchmark Results: 70% Less Network Traffic
The team ran three configurations against OTel demo traffic, ramping from 5,000 to 30,000 spans per second. No sampling and tail sampling each forwarded roughly 100% of input traffic. Retroactive sampling forwarded 70% less traffic after compression. Both tail sampling and retroactive sampling capped injections into the trace backend at about 1,000 spans per second, roughly 3% of input. The sampling rules were identical: keep traces over 5 seconds or with errors, sample 1% of healthy traces otherwise.
CPU, Memory, and Disk Trade-offs
Tail sampling used twice the CPU and memory of both the no-sampling baseline and retroactive sampling. Retroactive sampling saved 60 to 70% CPU and memory compared to tail sampling. Its CPU usage was slightly below even the no-sampling baseline because sending less data reduces encoding and decoding time. The cost is disk: retroactive sampling used up to 1.6 GB on the FIFO queue. Tail sampling held 4 GB in memory, so the swap is 4 GB of RAM for 1.6 GB of disk.
Complex Sampling Rules and Local Decision Making
Some sampling policies depend on dozens of span attributes, not just duration or error status. Sending all those attributes to the central server defeats the bandwidth goal. The solution is local decision making. If a local agent can evaluate a condition itself, such as a debug flag on a span, it marks that trace as sampled immediately and notifies the server to collect it from other agents. Traces the agent cannot decide on locally follow the standard retroactive path. No custom data structure extension is required.
Q&A
Does the sampling server use a standard OTLP protocol? The custom protocol applies only between local agents and the sampling server. After the decision, agents export full spans in standard OTLP to the trace backend. ▶ 27:57
Why a FIFO queue instead of a database? At consumption time the agent already has the decision, so it reads sequentially and checks headers. Spans with non-matching trace IDs are discarded without any random lookup. ▶ 30:01
Can the approach handle traces connected via span links rather than parent-child IDs? The prototype groups everything by trace ID, so span-link-connected traces with different IDs are not covered. The speakers noted the general idea of sending less data could still apply if a grouping mechanism exists. ▶ 32:37
Notable Quotes
retroactive sampling here as an as a comparison is sending 70% less traffic all of them up after the comparison after the compression. Zhu Jiekun · ▶ 16:47
tail sampling is using twice as much as CPU and memory than the other two, right? That means retroactive sampling saves 60 to 70% CPU and memory for you again. Zhu Jiekun · ▶ 17:43
we are trading 4 GB of memory with only 1.5 1.6 GB of disk space. I would say that’s a pretty good deal. Zhu Jiekun · ▶ 19:05
we don’t believe hotel is designed for efficiency, right? So, we trying to like uh implement only necessary fields out there in we compress them as well. Zhu Jiekun · ▶ 28:15
Key Takeaways
- Retroactive sampling cuts outbound tracing network traffic by 70% versus tail sampling at equivalent accuracy.
- CPU and memory drop 60 to 70% by buffering only key fields rather than full spans centrally.
- A FIFO queue with sequential IO avoids the random-seek overhead that plagues disk-offloaded tail sampling implementations.
About the Speaker(s)
Zhu Jiekun is an open-source software enthusiast and OpenTelemetry contributor at VictoriaMetrics, the company behind the high-performance metrics monitoring solution of the same name.
Roman Khavronenko is a software engineer with background in distributed systems, databases, and high-performance microservices. He has contributed to Prometheus, Grafana, and ClickHouse, and currently works on open source projects at VictoriaMetrics.