AI
Why Unstructured Logs Cost More Than They Reveal
Nearly every developer in the room had seen info-level logs with meaningless dashes and exceptions that say nothing. Those logs add noise, consume CPU, and cost storage money. Kernel-level trace points, Linux user events in BPF, and Windows Event Tracing for Windows already demonstrate how structured, low-overhead event logging works at scale. OpenTelemetry’s log work draws from those systems. Pająk has contributed to OpenTelemetry logs since November 2023 as the Go maintainer inside Splunk’s logging-focused team.
The OpenTelemetry Log Record Model
A log record in OpenTelemetry carries two timestamps (event time and observed time), trace correlation fields, an event name, a severity number, a human-readable body, attributes, resource attributes, and an instrumentation scope. Logs are not strings or raw JSON. They are structured domain objects. Recently, OpenTelemetry added support for complex attributes, including maps and heterogeneous arrays, shown through a GenAI semantic conventions example. Those complex attributes are now available across all signals in OTLP, eliminating the need to serialize data to JSON and deserialize it at the backend.
Events as Named, Versioned Log Records
The event name field distinguishes a well-known, versioned structure from generic bridged library output. When a record carries an event name, it signals a contract: specific attributes will be present, the structure is consistent, and semantic conventions govern behavior. Pająk points to the blog post “OpenTelemetry Logging You” as background for why events are defined as a special form of log record rather than a separate signal. The distinction matters for instrumentation library authors who previously had to depend on third-party libraries like log4j.
Deprecating Span Events in Favor of Log-Based Events
OpenTelemetry currently offers two ways to emit events: span events added to a tracing span, and events via the logs API. Two ways is one too many. The logs API path is richer, since it carries a severity level and exists independently of any span. Pająk announced that the span events API is being deprecated at the OTLP layer, not removed. Existing users get a migration path that either keeps events on the span or transfers them to log events. A blog post published roughly a week before the talk and a public GitHub issue track feedback on the migration.
The Enabled API and Performance at Scale
Previously, every log emission in OpenTelemetry allocated memory and consumed CPU even when no subscriber wanted the data. The new Enabled API lets code check first and only build expensive attributes when the record will actually be consumed. Pająk frames it as “measure twice, cut once.” The goal is near-zero allocation log emission when nothing is subscribed, matching the efficiency of ETW and Linux user events. The Enabled API was added to logs first, then extended to the metrics and tracing APIs.
Semantic Conventions: Consistent Naming Across Teams
Semantic conventions cover two areas. First, a registry of attributes, including cross-cutting fields like error type and service name, signal-specific fields like IO stream, and domain-specific semantics. Second, behavioral guidance: which severity levels to use, which attributes are recommended, how to report errors, and when to emit domain-specific events. Consistent naming lets separate teams share dashboards and correlate data across services without coordinating by hand. Pająk’s closing point: following semantic conventions makes your SREs, and your future self, happier.
Notable Quotes
logs, which are basically like trashy. So, they cause uh they cause noise They they add a lot of noise. They also uh they also affect our performance. Robert Pająk · ▶ 01:11
two ways is much too many. Robert Pająk · ▶ 10:07
it can be also useful for application developers that want to simply have a distinction between between logs that kind of are just like a debugging salt, or from something which actually matters and should, for example, fire alerts. Robert Pająk · ▶ 08:14
Now, go forth and log responsibly. Robert Pająk · ▶ 16:39
Key Takeaways
- Complex attributes (maps, arrays) are now available across all OpenTelemetry signals in OTLP.
- Span events are being deprecated at the OTLP layer; log-based events replace them with richer data.
- The new Enabled API cuts wasted allocations by checking subscriber interest before building attributes.
About the Speaker(s)
Robert Pająk is a Software Engineer at Splunk, a Cisco company, and an OpenTelemetry Go maintainer. He has contributed to OpenTelemetry logs since November 2023 and works in the log Special Interest Group. His background spans software design, concurrent programming, relational databases, cyber security, performance tuning, testing, automation, and observability.