AI
The Kernel/User-Space Boundary Every Agent Developer Must Understand
Every telemetry agent runs in user space. The kernel owns the hardware. The only bridge between them is system calls. System calls are expensive: each one typically copies the memory buffer rather than passing a reference, so the data is duplicated before any network operation begins. Abusing system calls drives up CPU time. Understanding this boundary, even without writing kernel code, gives engineers a concrete mental model for diagnosing why data movement is slower than expected.
Why MessagePack Replaces JSON Inside the Pipeline
Fluent Bit normalizes all incoming data into MessagePack, a binary format that has been around for 12 to 13 years. The same message that costs 27 bytes in JSON costs 18 bytes in MessagePack, with no compression applied. Boolean values become single codes like C3 instead of ASCII characters. Boolean maps carry compact type codes instead of text keys. MessagePack is schema-less, unlike Protobuf. These internal chunks can live in memory or on the file system. Before delivery, the agent converts the chunk back into whatever format the destination expects, such as JSON for Elasticsearch.
Batching and Buffering: Fewer Operations, Less Contention
Sending each chunk with its own system call is expensive. Network cards buffer at the hardware level for the same reason: many small writes multiply copy operations, system call overhead, and serialization cost. The answer is batching. Silva compared it to doing five bank errands on five separate days versus finishing all five on Monday. The same logic applies to the CPU. Reducing the number of write operations lowers both system call frequency and context-switching cost, freeing CPU cycles for actual data processing.
Event Loops and Coroutines: Async I/O at Scale
Fluent Bit separates concerns across threads. One thread runs the main event loop: it buffers incoming data, runs the scheduler, and handles retries. A second thread sends data. Adding filters in the main thread creates contention as volume grows. Moving processing into separate processor threads lets the agent use additional CPU cores. Coroutines, lightweight user-space threads, make this practical. When a connection handshake blocks, the coroutine suspends and the event loop resumes it once the kernel signals readiness. From the developer’s perspective the code reads sequentially; underneath, execution pauses and resumes around every slow I/O call.
Fluent Bit Architecture: Ten Years of These Principles
The architecture Silva described, async sockets, event loops, coroutines, and multi-threading, mirrors what Nginx introduced after Apache’s one-process-per-connection model proved too expensive. Apache V1 forked a new process for each HTTP request. Apache V2 moved to threads. Nginx added async I/O. Fluent Bit followed a similar path over its 10-year history. The project runs on three principles: high performance, low resource usage, and vendor neutrality. Telemetry data movement defines performance. Buffering defines stability. Concurrency defines scale.
Notable Quotes
message pack is like a binary JSON that has been around for 12 13 years and basically it’s a binary format in JSON for example the same message will take 27 bytes but in message pack will be 18 and there’s no compression here Eduardo Silva · ▶ 10:37
telemetry system are about data movement and data movement defines performance buffering defines stability ility and concurrency also escalability Eduardo Silva · ▶ 23:24
if you mix all these concepts with multi-threading with cor routines with event loops, you can move the data at a very very high scale Eduardo Silva · ▶ 22:43
our applications sometimes abuse of system calls right if you use more most system calls sometimes it gets more expensive at a in terms of CPU time Eduardo Silva · ▶ 04:44
Key Takeaways
- MessagePack cuts the same message from 27 bytes to 18 bytes with no compression.
- Each system call copies memory buffers, making per-message writes expensive at scale.
- Coroutines suspend blocking I/O calls so the event loop can service other work immediately.
About the Speaker(s)
Eduardo Silva is a Distinguished Engineer at Chronosphere, a Palo Alto Networks Company. He is one of the maintainers of the Fluentd project and the creator of Fluent Bit, a lightweight processor for logs, metrics, and traces. He has worked in the Fluent ecosystem for nearly 10 years.