The Gap Between How Telemetry Is Built and How It Should Work

▶ Watch (0:08)

Most teams add metrics during an incident, copy patterns from another service, or discover hundreds of unexpected metrics after importing a new dependency. Nobody writes the schema first. OpenTelemetry Weaver is built for the opposite approach: treat telemetry like an API, declare it in a YAML registry, and generate everything else from that single source of truth. Shopify does this with a team of 20 engineers focused solely on observability. Most organizations do not have that option.

What Weaver Does: Schema as the Source of Truth

▶ Watch (0:08)

Weaver is an OpenTelemetry project that takes a registry YAML as input and produces artifacts from it. A single entry, such as http_request_duration_seconds with a description, units, and examples, can drive code generation, documentation, dashboard templates, and policy enforcement. Prometheus naming conventions require the unit as a metric name suffix. Weaver can enforce that rule org-wide. The same registry can output instrumentation for both the Prometheus Go client and the OpenTelemetry SDK simultaneously, using Jinja templates.

Demo: Inferring a Schema from Live Telemetry

▶ Watch (6:42)

The demo started with a small Go application exporting two metrics, http_requests_total and parse_souls_total, via the Prometheus client. The make infer-schema command spun up an OTel Collector configured with a Prometheus receiver, scraped the application, translated the output to OTLP, and pushed it to Weaver. A load generator sent varied HTTP inputs (200s, 500s, 429s) to capture all label combinations. Weaver produced a populated registry YAML covering every metric and attribute the application actually emitted. Both metrics appeared in the generated file.

Contract Testing with Live Check

▶ Watch (15:15)

weaver registry live check receives an OTLP payload and compares it against the registry YAML. Running it immediately after inference produced zero violations. Then the demo changed the metric unit from seconds to milliseconds in the Go source without updating the registry. Live check flagged a violation: http_milliseconds did not exist in the registry. This check runs in CI via GitHub Actions and blocks pull requests. A single-line metric rename buried in hundreds of changed lines, introduced by a code editor prompt unnoticed in review, would be caught before merge.

Code Generation and the Path to Schema-Driven Instrumentation

▶ Watch (18:18)

Once the registry exists and live check passes, weaver generate produces typed client SDKs from Jinja templates. The demo generated Go clients for both Prometheus and OpenTelemetry into a generated/ folder. Changing the registry entry from seconds to milliseconds and re-running make generate produced compilation errors in the application code, surfacing every call site that needed updating. Client libraries can be published to a package manager so downstream consumers import a versioned, schema-derived package rather than writing instrumentation by hand.

Prometheus Schema Awareness and the Adoption Roadmap

▶ Watch (24:40)

The Prometheus team is experimenting with making Prometheus schema-aware. If a schema documents a rename from http_request_duration_seconds to milliseconds, Prometheus will merge both series at query time. Querying the new name returns old data; querying the old name returns new data. Bartek presented the underlying approach at KubeCon Paris 2024. The recommended adoption path is: run inference against staging or production traffic, add live check to CI, then shift to generated clients so the schema becomes the only place a metric name can change.

Q&A

Can a shared base registry be imported to avoid duplicating standard metrics across many repositories? The team confirmed this is on the roadmap: a future infer flag will accept an existing registry as input and output only the diff, so teams can converge toward standard definitions incrementally. ▶ 27:31

How do teams propagate a base metric change once generated client libraries are already distributed? The schema must become the source of truth org-wide; CI should block manually written instrumentation so every client regenerates from the schema on change. ▶ 29:25

Are there templates or documentation guidance for Weaver’s Jinja templates? The Weaver maintainers created a new weaver-packages repository and plan to distribute default templates inside the CLI container image. ▶ 29:39

Can Weaver enforce standard metric naming like OpenTelemetry semantic conventions across divergent teams? Once auto-generated SDKs are adopted, the schema controls the metric name; templates can emit both old and new attribute names in parallel during a gradual rollout, then drop the old one after consumers migrate. ▶ 30:52

Notable Quotes

Observability by Design is the dream, but in reality, it’s not how things really work. We get into them in our org by we have an incident and then we desperately create a new metric because we’ll need in the future Arthur Silva Sens · ▶ 05:00

you don’t have like alerts firing, you don’t have dashboards working, or even worse, you don’t have systems like KEDA not scaling your applications and causing a a potential production outage, right? So, this very dangerous. Nicolas Takashi · ▶ 17:29

the schema needs to be the source of truth and you need [snorts] to roll this out through your organization. Once everything is based on the schema, it’s impossible to like the idea is that a CI should block anything that is manually written. Arthur Silva Sens · ▶ 29:24

I this I feel like this infer command is something that the industry industry needs to [snorts] adopt observability by design. Arthur Silva Sens · ▶ 25:47

Key Takeaways

  • weaver registry infer generates a registry YAML from live OTLP traffic, capturing every real label combination.
  • weaver registry live check runs in CI and blocks any pull request that changes a metric without updating the schema.
  • Typed generated clients surface metric renames as compilation errors, removing the need for manual instrumentation.

About the Speaker(s)

Arthur Silva Sens is a Software Engineer at Grafana Labs focusing on Prometheus and OpenTelemetry interoperability. He leads the Prometheus Special Interest Group in the OpenTelemetry community alongside David Ashpole, and implemented the weaver registry infer command by reading the Rust book from scratch.

Nicolas Takashi is the observability stack tech lead at Coralogix and a maintainer of Perses. He contributes to Thanos, Prometheus, and OpenTelemetry at the CNCF, and drove the live demo showing schema inference, contract testing, and code generation end to end.