AI
The Spanimals: Animal Mascots for Observability Signals
Before touching a single dashboard, Jernigan and Haeussler introduce five animal characters. Tracy makes the end-to-end jump across a distributed request; the spannies alongside her are the spans within that trace. Logy, a parrot, represents logs: chatty, formatted differently per service, and worth filtering. Metric, an elephant with a large memory, tracks numbers, gauges, and rates. Flamy, named after flame graphs, handles profiling. Collie collects and groups all four, mapping to the OpenTelemetry collector.
The Demo App: A Polyglot Distributed System
The demo has four components: a runner, a fact generator, an image generator, and a Java/Spring image backend. Pass it an animal name and the fact generator calls OpenAI for a fact. That fact feeds the image generator, which also calls OpenAI, then stores the result in PostgreSQL via the Java backend. The other three services run Python. The split was intentional: a polyglot, distributed app generates richer telemetry and shows how signals correlate across language boundaries.
Traces and Spans: Reading a Distributed Request
In Grafana Drilldown, a 23.6-second trace spans the entire request. Each smaller rectangle inside is a span. The runner calls the fact generator and image generator separately, so those spans appear at the same level. The image generator then calls the image backend, which appears nested beneath. Span attributes show the language (Java), the database operation (INSERT into PostgreSQL), and timing. Clicking a span opens its logs, correlating what was logged at exactly that point in the call.
Signal Correlation: Tracing the Goat Delay
Haeussler injected a deliberate 5-second delay into any request for a goat. In the trace view, the runner component, which normally takes milliseconds, suddenly shows a large percentage of the total call time. Clicking that span and requesting its logs surfaces a message asking whether the user really wants a goat fact. The error case, a fake animal named stomal fox, shows an exception immediately. It never reaches the fact generator. The runner fails first.
Metrics, Flame Graphs, and Collector Flexibility
The metrics view shows 50 collected metrics. Filtering to “DB” gives 13 database metrics. Filtering to Java surfaces garbage collection time, memory consumption, and invocation rate. Profiling goes further. The eBPF profiler in Grafana Pyroscope works across both Java and Python. A flame graph shows how much CPU time each call stack consumes, and some tools can link that back to a specific line of GitHub source. The OpenTelemetry collector config routes each signal separately: one config sends traces to Tempo, metrics to Prometheus, logs to Loki, and profiles to Pyroscope.
Notable Quotes
this is not a setup this is really Matthias Haeussler · ▶ 14:16
Demos are hard. Tiffany Jernigan · ▶ 15:11
it is a lot newer than the other signals Tiffany Jernigan · ▶ 25:23
Key Takeaways
- OpenTelemetry’s four signals (traces, logs, metrics, profiles) each answer different questions about application behavior.
- The OTel collector routes all four signals to vendor-swappable backends without changing application code.
- Clicking a span and jumping to its logs at that timestamp cuts root cause analysis from minutes to seconds.
- Flame graphs show CPU time per call stack and can link specific hot spots back to GitHub source lines.
About the Speaker(s)
Matthias Haeussler is VP Expert at CGI, a university lecturer for distributed systems, Linux Foundation trainer, Docker Captain, and Cloud Foundry ambassador. He organizes meetups and focuses on enabling teams across cloud-native strategy and technology.
Tiffany Jernigan is senior developer advocate at Grafana Labs and a CNCF Ambassador. She previously worked as a software developer and advocate at VMware, Amazon, Docker, and Intel, and holds an electrical engineering degree from Georgia Tech.