AI
Six Types of Telemetry Enrichment
Raw signals capture a moment in time, but they often lack context for SRE practitioners or cost-center attribution. Duarte laid out six enrichment categories: reshaping existing data, adding infrastructure metadata, static dictionary lookups, remote or large external source lookups, event correlation, and model inference such as anomaly detection or scoring. The OTel Collector handles the first two well today. Types three through six exposed a gap that drove two to three years of GitHub issues requesting a lookup capability.
Why Inline Config Breaks at Scale
Type three enrichment, translating IDs to names using a dictionary, currently forces users to embed every key-value pair directly in their collector config. That works at 50 entries. At 50,000 it becomes unmanageable. The same problem repeats for external or frequently changing sources such as threat feeds, which are too large to scrape into a static file. These limitations produced the recurring community proposals, ranging from source-specific processors to a generic gRPC processor, that Duarte consolidated into a single lookup processor proposal.
The Lookup Processor: Design and Configuration
Duarte submitted the proposal in August 2024. The collector SIG accepted it in December 2024, with co-sponsorship from Sam at Grafana. The processor queries a source using a key, retrieves one value or a map of values, and writes results into signal attributes. The key can be a plain attribute path or a full OTTL expression for pre-processing. Lookups run at either the record or resource level. Built-in sources ship with the processor; vendor-specific sources plug in through a small factory and source interface, kept inside a custom collector distribution.
Live Demo: Four Enrichment Layers on One Log
The demo ran a single collector on minikube with telemetry-gen emitting one log every 5 seconds. Starting from a raw log with a JSON body and one app.server attribute, Duarte applied four processors in sequence. The transform processor parsed the body into discrete attributes including source.ip and request.id. The Kubernetes attributes processor added pod name, namespace, and cluster. The YAML-source lookup returned a risk score of “high” and a category of “trap” for the IP. The DNS pointer lookup then resolved that IP to a hostname.
Current State and What Comes Next
The base processor merged to main in late February 2025 with YAML source support for logs only. A second open PR adds the DNS source with pointer queries, a configurable DNS server, and negative caching enabled by default. Planned additions include an HTTP source, DNS A and AAAA record types, multi-server DNS support, YAML hot-reloading, traces and metrics signal support, and built-in cache hit and miss rate telemetry. Both existing sources already carry benchmark suites. The next milestone is promotion to alpha and inclusion in the OTel Collector contrib distribution.
Q&A
Can teams add their own lookup sources without contributing to contrib? Yes. The factory and source interfaces allow custom sources to live in a private repository and be registered when the processor is instantiated, staying entirely within a custom collector distribution. ▶ 20:58
How does a slow or unavailable external source affect the pipeline? Configurable timeouts limit how long a lookup blocks, and failed DNS lookups are cached as negatives so the resolver is not hit again immediately. Signals that cannot be enriched pass through unenriched, or the operator can choose to hold them. ▶ 22:37
Notable Quotes
that works well if you have 50, but if you have 50,000 or more then becomes troublesome. João Duarte · ▶ 05:51
this is something that we as Elastic that I work for didn’t want to fully own. I think this is important enough and widely impactful enough that we would want to have more voices and more people from other companies to do this work with us. João Duarte · ▶ 07:44
Of course, it needs to be fast otherwise it will impact your performance in your uh pipelines. João Duarte · ▶ 17:57
Key Takeaways
- The lookup processor proposal was accepted in December 2024 and merged to main in late February 2025.
- YAML and DNS sources ship first; custom sources plug in via two small Go interfaces without touching contrib.
- Cache hit and miss rate telemetry will be added so operators can tune the processor without guessing.
About the Speaker(s)
João Duarte is a Principal Software Engineer and Tech Lead for Logstash at Elastic, focused on large-scale data ingestion, observability, and performance. He contributes to the OpenTelemetry Collector, working to extend its capabilities to meet practical production requirements.