How the OTel Collector Sees Your Data

▶ Watch (1:16)

OpenTelemetry structures telemetry into signals (logs, metrics, traces), each with a resource, instrumentation scopes, and items. The resource holds attributes like service.name, set once at startup. Each instrumentation scope holds its own slice of log records, metric data points, or spans. When data reaches the OTel Collector, every receiver translates its payload into pdata, the collector’s internal representation of the protobuf model. OTTL only works with pdata. It cannot see receiver payloads or export-formatted data. Debugging starts by understanding how the collector sees your data, not how your backend displays it.

The OTTL Debug Toolkit

▶ Watch (6:01)

Three tools cover most OTTL debugging. The debug exporter, configured with verbosity: detail, prints full telemetry to the console: field names, attribute types, and actual values. It shows the end state of a pipeline but not how data moved through it. For step-by-step tracing, set service.telemetry.log.level: debug. Those logs emit a transform context snapshot before any statement runs, then a follow-up after each statement showing the condition result and what changed. The OTTL playground runs the same collector code compiled to WebAssembly in a browser, with visual diffs and debug logs, no local collector required.

Checking Assumptions About Data Shape

▶ Watch (11:39)

A log transform sets severity to 17 (error) when the body contains the string “error”. Two logs should match; only one does. The debug logs show the condition evaluated to false on the second log. The transform context reveals the body contains lowercase “error” while the statement matched against uppercase “ERROR”. The collector stores exactly what the receiver delivered, not what a backend might display. The fix is case-insensitive regex matching. Both logs then match and both severities update correctly.

Tracing Statement Execution Step by Step

▶ Watch (14:56)

A log body holds a JSON string with an elapsed time field. The goal: parse that JSON and replace the log body with the elapsed time value. The statement parses the body into the cache under key “body”, then reads cache["body"] to set the log body. The result is a complex map, not a string. The playground’s debug logs show the cache has one entry: key “body”, value is the full parsed JSON map. Reading it returns the map, not the nested field. The fix: assign the parsed JSON directly to the cache root.

Shared Pdata Fields and Last-Write-Wins

▶ Watch (17:50)

A statement copies the server address attribute from a log record up to its resource. The visual diff shows the new resource attribute with an IP address. But in pdata, one resource can serve multiple logs. If two logs under the same resource have different server address values, a set on the resource runs once per log iteration, with the last write winning. The value looks random but is deterministic: whichever log processes last sets the final value. Switching to append reveals both IP addresses. Any statement that promotes attributes up the pdata hierarchy faces this same behavior.

Context Inference and Performance

▶ Watch (20:12)

A metric transform adds a metadata tag to histogram metrics, filtered by metric type. The output looks correct. But the debug logs show hundreds of “after statement execution” entries for just two metrics. The data point context iterates every data point in every metric before running the statement. None of the paths reference data point fields, making this unnecessary work. Removing the explicit context config enables context inference: OTTL examines the statement’s paths and selects the tightest match. With metric context, the debug logs drop to two entries. Execution time drops from several milliseconds to two or three.

Q&A

Can OTTL transformations be unit-tested in a CI/CD pipeline? The OTTL package is available in contrib and importable outside the collector, though any input must be shaped as pdata. Nobody on the team has used it for CI/CD pipeline testing specifically, but the library is there to try. ▶ 25:33

Is there a way to enable debug logs for only one transform processor, not the whole collector? The collector is all-or-nothing today, with no per-component debug log mechanism. A component-specific logger is technically possible in Go but would need to avoid violating other expectations users have for collector telemetry export. ▶ 26:59

Notable Quotes

actually have? Lowercase error. Tyler Helmuth · ▶ 14:08

one that persists. The last one wins and Edmo Vamerlatti Costa · ▶ 19:37

The collector uh is all or nothing. Edmo Vamerlatti Costa · ▶ 27:12

Key Takeaways

  • OTTL works on P data. Inspect what the collector actually sees before writing any statement.
  • Debug exporter shows end-state; debug logs show step-by-step condition evaluation and field changes.
  • Let context inference choose the tightest context. Wrong context multiplies iterations unnecessarily.

About the Speakers

Edmo Vamerlatti Costa is a Principal Software Engineer at Elastic and an OpenTelemetry Collector contributor who helps maintain the OpenTelemetry Transformation Language.

Tyler Helmuth is a Staff Software Engineer at Honeycomb and a maintainer for the OpenTelemetry Collector and Helm Charts, with contributions across multiple OTel repositories.