What Fluent Bit Does and Why It Exists

▶ Watch (0:19)

Fluent Bit is a telemetry pipeline. It collects logs, metrics, and traces from hosts and applications through input plugins, processes that data (removing duplicates or obfuscating for security), and routes it to backends like Splunk, Google Cloud Logging, or Elasticsearch. The project has run for 10 years on a single constraint: high throughput with low resource use. It runs on IoT devices, Linux nodes, bare-metal machines, and as a standalone aggregation gateway.

Fluent Bit V5: OpenTelemetry and Metrics Processing

▶ Watch (2:29)

V5 updates OpenTelemetry protocol support to version 1.10, which adds profiling in alpha. The new cumulative-to-delta processor lets operators collect only the difference between metric counter samples rather than the full cumulative value. A new topological data analysis processor ships alongside improved trace sampling. V5 now supports both head and tail sampling modes, matching what the OpenTelemetry Collector offers, and adds processing of all auto-log attributes.

Security and HTTP Scaling

▶ Watch (3:36)

The internal HTTP server was rearchitected to accept connections and scale processing across all pipeline layers. On the security side, V5 implements OAuth2 as a generic agent-level mechanism. Any pipeline component can use it to fetch secrets and share them across the full pipeline. Previous versions required per-component credential configuration. This change reduces the surface area for misconfiguration when secrets need to reach multiple inputs or outputs.

Performance: V5 vs V4.2 vs OpenTelemetry Collector

▶ Watch (4:00)

Silva ran a baseline benchmark comparing Fluent Bit V4.2, Fluent Bit V5, and the OpenTelemetry Collector at their latest default configurations. V5 exceeded V4.2 on OTLP payload throughput. On CPU utilization, V5 (red) came in lower than both V4.2 (blue/purple) and the collector (green). Memory followed the same pattern. Every CPU cycle saved translates to a smaller cloud bill, which Silva cited as the direct reason the team tracks these numbers.

Notable Quotes

nobody wakes up a morning saying I want to do telemetry observability. Eduardo Silva · ▶ 00:25

every optimization that we do we might save a few cents per CPU cycles Eduardo Silva · ▶ 04:13

fluent bit 5 well exceeded fluent bit 42 Eduardo Silva · ▶ 04:36

Key Takeaways

  • Fluent Bit V5 ships OpenTelemetry protocol 1.10 with alpha profiling support.
  • A new cumulative-to-delta processor reduces metric volume at collection time.
  • V5 outperforms both V4.2 and the OpenTelemetry Collector on CPU and memory in default benchmarks.

About the Speaker(s)

Eduardo Silva is a Distinguished Engineer at Chronosphere (a Palo Alto Networks company). He created Fluent Bit and maintains the Fluentd project. His work focuses on lightweight processors for logs, metrics, and traces.