AI
Fluent Bit as a Full Data Plane, Not Just a Node Agent
Fluent Bit started as a lightweight node agent designed to forward data to Fluentd. V5 changes that positioning. Silva argues that 2026 is the right time to reclaim the aggregator role. The project now targets full data plane use: collection, processing, sampling, and aggregation in a single binary. The constraint is unchanged: performance and low resource use stay non-negotiable. Building something slow defeats the purpose of replacing a heavier aggregator.
Conditional Routing and Processor Enhancements in V5
Processors in YAML configuration attach directly to an input plugin rather than relying on tag matching across the whole pipeline. V5 adds routing with conditionals inside the same input block. After processing runs, an optional routing section can send data to specific destinations based on key values or patterns. Previously achieving this required rewrite-tag rules and additional input plugins. The new approach eliminates that complexity. A separate cumulative-to-delta metric processor is also new, the first processor in Fluent Bit that generates a new sample rather than just filtering existing data.
Multi-Threading and the Unified HTTP Server
Fluent Bit historically handled all inbound HTTP traffic on a single worker thread. V5 unifies the internal HTTP server API and adds multi-threading support. The Splunk input plugin, for example, can now spawn 10 workers. HTTP, Elastic Search, OpenTelemetry, and Prometheus Remote inputs all benefit. The unification also removed more than 4,000 lines of duplicated code that had accumulated from separate HTTP/1 and HTTP/2 handling paths. Internal logs can now route through the pipeline via a new fluent-bit-logs plugin, spawning a dedicated logger thread.
OAuth2 and Security for Enterprise Deployments
Financial institutions and banks require more than static tokens. V5 adds a full OAuth2 implementation supporting both client and server roles. A Fluent Bit agent retrieves a token from the OAuth2 server, attaches it to outbound requests, and the receiving Fluent Bit instance validates that token against the same server before accepting data. To keep per-request validation overhead low, the server side uses a hash table cache keyed on token expiration. When a token approaches expiry, the cache entry is renewed. The implementation is agnostic at the API level, so HTTP and OpenTelemetry plugins use it without plugin-specific code.
V5 Performance: OTLP Throughput and CPU Usage
Silva benchmarked V5 against Fluent Bit 4.2 and the OpenTelemetry Collector using OTLP ingestion with no processing. V5 delivers higher maximum throughput than both. The tradeoff is CPU: V5 uses more than 4.2, but stayed under 150% CPU (1.5 cores) with four workers configured and two actually used. Memory usage is comparable across versions. Trace sampling is 10% faster due to reduced data copies during the correlation window before sampling decisions are made. Silva recommends running your own benchmarks against your specific workload before migrating.
Q&A
How does Fluent Bit validate JWT tokens from OAuth2, and what happens to performance? Fluent Bit validates tokens by calling back to the OAuth2 server, caches results in a hash table keyed by token expiration, and renews entries before they expire to avoid per-request round trips. ▶ 24:05
Can Fluent Bit V5 replace the OpenTelemetry Collector? Silva says Fluent Bit integrates with OpenTelemetry rather than displacing it. Many users run Fluent Bit alongside the Collector, offloading heavy processing to Fluent Bit before forwarding. ▶ 25:01
Do existing filters still work in V5, or do they need to be rewritten as processors? Existing filters are backwards compatible and can attach as processors without changes. The project avoids deprecating plugins because it has no visibility into who is using them. ▶ 38:07
Should teams replace Fluentd with Fluent Bit V5 for aggregation? If Fluentd is used only as an aggregator without Ruby gem processors, Silva says Fluent Bit has evolved significantly more in recent years. Financial institutions and Google and Amazon have already migrated. ▶ 36:40
Notable Quotes
nobody want to wake up in the morning and say, “Oh, I’m going to do observability.” That’s boring, right? You want to do data analysis. Eduardo Silva · ▶ 01:09
also we were able to remove more than 4,000 lines of duplicated code because before in order to handle HTTP 1, HTTP2, we we had a lot of redundancy in the API Eduardo Silva · ▶ 15:14
I always say it’s not the same to run a cast very very fast than running a car very very fast, making sure that you you optimizing, you know, the fuel for that car Eduardo Silva · ▶ 20:35
Yeah, we need help. That’s » not going to lie. Eduardo Silva · ▶ 43:35
Key Takeaways
- V5 positions Fluent Bit as a full data plane, replacing the aggregator role previously held by Fluentd.
- Multi-threaded HTTP ingestion and a unified server API removed over 4,000 lines of duplicated code.
- Trace sampling runs 10% faster in V5 through reduced memory copies during the correlation window.
- OAuth2 support with server-side token caching meets strict security requirements from financial institutions.
- Existing filters attach as processors without changes, preserving backwards compatibility across all configurations.
About the Speaker(s)
Eduardo Silva is a Distinguished Engineer at Chronosphere, a Palo Alto Networks Company. He is one of the Fluentd project maintainers and the creator of Fluent Bit, a lightweight Logs, Metrics, and Traces processor.