AI
Head-Based vs Tail-Based Sampling
Head-based sampling decides at the first service and propagates via the traceparent header. All downstream services respect that decision. It costs no extra memory but can drop error traces decided before the failure happens. Tail-based sampling buffers all spans in the collector, grouped by trace ID, waits up to 30 seconds, then evaluates the full trace against error or latency policies. It catches what head-based misses but needs more memory and delays query visibility.
Telemetry Profiling with the Count Connector
The count connector counts spans, logs, and data points, splits counts by service name and Kubernetes namespace, and exports them as metrics to Prometheus. In the demo cluster, the frontend service sent several times more spans than the three backend services combined. Backend-3 sent far more logs and debug-level entries than the others. Health-check endpoints generated spans from all services, with the frontend producing five spans per health invocation, none useful for troubleshooting.
Filtering Noisy Spans and Logs
The filter processor drops data the collector should never forward. Debug logs from backend-3 and log records with bodies over 1,000 bytes go first. Health-check spans drop on an HTTP root attribute match. But filtering a trace’s root span leaves orphan child spans behind, harder to find than the originals. The only reliable fix is disabling sampling for health endpoints in the OpenTelemetry SDK config, which marks all child spans not-sampled before they leave the application.
Head-Based Sampling: SDK Config and Collector Processors
The instrumentation CR configures sampling via environment variables injected at pod start. Changing from 100% to 50% requires restarting every deployment to pick up the new config. The Jaeger remote sampler offers per-endpoint and per-service configuration from a centralized collector component but fully supports only Java and Go. The probabilistic sampler processor on the collector provides a safety net without touching the application, with equalizing mode capping ingestion at a maximum percentage regardless of what the SDKs sent.
Tail-Based Sampling: Policy Design and Memory Planning
The tail sampling processor waits 10 seconds before evaluating each buffered trace. This demo’s policies keep all error traces, keep traces over 2 seconds, drop health-check traces entirely, and sample the rest at 10%. Memory planning: at 50 requests per second with a 10-second wait and a 2 KB average span size, the collector needs roughly 15 MB. At 1,000 spans per second that grows to 600 MB. Scaling requires a load balancing exporter to pin spans from the same trace to the same collector instance.
Log Deduplication at the Collector
The log deduplication processor groups logs by body content or attributes within a time window, forwards one copy, and emits a metric with a duplicate-count attribute. Loffay noted this cuts log storage costs without any application code change. The tutorial cluster had no duplicates in the demo, but the processor ships ready to apply. Collector-internal telemetry exposes the deduplication rate and count so operators can verify it is working. The processor slots into the log pipeline in the collector config alongside the filter and sampling processors.
Notable Quotes
sampling is a technique how we actually decide which data we want to store. It’s not necessary to store all the data uh because hopefully most of the telemetry data we never touch right. Pavol Loffay · ▶ 8:19
so be very careful when you filter the the health endpoints. Pavol Loffay · ▶ 36:48
the skill would reconfigure the collector for the most common use cases or help you to understand which is like what malicious data you have on the collector and based on that uh apply the the right configuration to automate all this process because it takes a lot of time to to figure out what is actually malicious in your data Pavol Loffay · ▶ 1:08:09
Key Takeaways
- Profile telemetry by service before sampling to identify which workloads drive disproportionate data volume.
- Filtering health-check spans at the collector root leaves orphan child spans that are nearly impossible to locate.
- Tail-based sampling at 1,000 spans per second with a 10-second wait needs roughly 600 MB of collector memory.
About the Speaker(s)
Pavol Loffay is a principal software engineer at Red Hat working on open-source observability technology for modern cloud-native applications. He contributes to and maintains CNCF projects OpenTelemetry and Jaeger.