AI
Why Centralized Storage Fails Distributed Applications
NAVER’s storage platform runs about 5,000 nodes, 50,000 disks, and serves 250,000 persistent volumes at roughly 100 petabytes of capacity. The base platform uses Ceph, which applies three-way replication or erasure coding at 1.5x redundancy. Distributed applications like Kafka and Elasticsearch add their own replication on top, typically two or three copies. Stacked together, the same data can be stored up to nine times. That multiplier hits IO directly: more copies means more IO on every write.
Five Goals That Shaped the Hyperconverged Design
NAVER defined five requirements before choosing any technology. First, replication must be flexible: apps that already replicate their own data need only one storage replica, while others need storage-level redundancy. Second, workload isolation should improve so a single noisy neighbor or disk rebuild does not ripple across the whole cluster. Third, storage should sit on the same node as its workload to minimize latency, but fall back to remote access when local capacity runs out. Fourth and fifth, pod and storage scheduling must stay coordinated, and replicas must be movable for maintenance.
Longhorn Architecture: Engines, Replicas, and CSI
Longhorn maps each Kubernetes PersistentVolume to one Longhorn volume. That volume has two components: an engine and replicas. The engine sits in the data path, processes reads and writes from the application, and keeps replicas consistent. The replicas store the actual data and can be spread across different nodes for redundancy. From a user perspective it looks like any standard StorageClass. Longhorn exposes three data-locality options in the StorageClass: disabled, best-effort, and strict-local. NAVER chose best-effort, which places one replica on the workload node during normal operation but allows movement during maintenance.
CSI Storage Capacity: Fixing Initial Placement
Without storage-capacity awareness, the Kubernetes scheduler places pods based on compute resources alone. A StatefulSet with two pods, each requesting a 40 GB PVC, could land both pods on nodes whose storage is already exhausted, forcing replicas to sit elsewhere. CSI Storage Capacity (introduced by KEP-1472) lets the scheduler filter nodes before placement. Longhorn’s GetCapacity implementation checks each disk, excludes unhealthy or unschedulable disks, and returns the maximum single-disk schedulable capacity rather than the sum. That prevents Kubernetes from scheduling a PVC larger than any single disk can hold.
Where CSI Storage Capacity Falls Short
CSI Storage Capacity only applies during the first scheduling of a pod, before the PVC is bound. Once the volume is bound, later rescheduling events such as pod restarts or node drains skip the storage capacity check entirely. For NAVER, this matters because replicas are expected to follow pod placement. A pod restarted on a different node may land somewhere without a local replica, triggering an unnecessary rebuild. The scheduler also cannot see where existing replicas already live, so it cannot prefer nodes that already hold the data.
A Custom Scheduler Plugin for Storage-Aware Placement
To close the gap, NAVER built a custom scheduler plugin using the Kubernetes scheduler framework. The plugin adds Longhorn state directly into pod scheduling. On a pod restart, it first checks whether an existing replica already lives on a candidate node and prefers that node. If no suitable replica node is schedulable, it falls back to filtering by available storage capacity. The plugin operates at the filter, score, reserve, and bind extension points. It is still in internal testing. NAVER plans to publish implementation details once the service proves stable in production.
Notable Quotes
In the worst case, the same data stored up to nine time. It also affect performance through IO amplification. Jangseon Ryu · ▶ 03:50
and importantly it returns the maximum among the disk not the sum. Uh this is intentional because a single PBC cannot be cannot be split across multiple disk. Jinhong Kim · ▶ 21:04
what we needed was not just capacity aware scheduling but longhorn aware or more broadly storage aware scheduling. Jinhong Kim · ▶ 25:41
Key Takeaways
- Stacking Ceph replication on top of app-level replication can store data up to nine times.
- Longhorn’s best-effort locality keeps replicas on the workload node without hard-locking storage to one node.
- CSI Storage Capacity solves initial pod placement but does not help after a PVC is already bound.
- A custom scheduler plugin is required to prefer nodes that already hold an existing replica.
- NAVER operates 250,000 persistent volumes across 50,000 disks and is still testing the custom scheduler internally.
About the Speakers
Jangseon Ryu is a Senior Technical Lead at NAVER. He builds and operates large-scale storage platforms that support cloud and container-based services. With over 20 years of experience in cloud infrastructure, he has worked extensively with OpenStack, Ceph, and Kubernetes, architecting and maintaining hundreds of production-grade storage clusters.
Jinhong Kim is a Software Engineer at NAVER focused on storage for containerized environments. His work spans the full storage stack, from flash devices and kernel drivers to large-scale distributed storage systems. He currently operates large-scale storage clusters and contributes to open source storage projects.