AI
Why the Current TSDB Format Slows Thanos Down
Four open issues in the Thanos tracker frame the problem. The store is unpredictably slow depending on query shape and data layout. The offset and symbol tables hit a four-byte size limit, so the compactor uses heuristics to drop blocks from compaction groups, meaning some blocks never get compacted. Sizing the store gateway is nearly impossible, which leads teams to overprovision. At startup, building the binary index header consumes enough memory to trigger out-of-memory kills.
How Parquet Fixes the Random-Read Problem
The TSDB format scatters postings and series definitions across an index file. Fetching series 1-10 and then series 1,000-1,010 requires multiple separate object storage requests. Every object storage implementation forces one request per byte range. Parquet organizes data into pages with per-page statistics. A query with a matcher for a specific label value can skip entire pages that provably contain no match, reducing the number of requests. Column orientation means a sum-by query fetches only the three or four relevant label columns instead of all of them.
Storage Savings Measured at Vinted
Giedrius converted several production tenants. A small tenant with no churn shrank 85%. A tenant storing node exporter recording rule results, which has moderate churn from changing network interface names, shrank 42%. A tenant with server recording and alerting rules shrank 60%. Vinted’s largest region currently holds over 700 terabytes. At the worst-case 42% reduction, that drops to roughly 400 terabytes. That is 300 terabytes less to store with no loss in precision.
Query Speed in Production: 5x With Three Matchers
Benchmarks ran on a physical server using Grafana k6 with 50 iterations, all caches disabled, against Vinted’s production Kubernetes-hosted Cortex deployment. The simplest possible query, equivalent to SELECT *, already runs faster under Parquet because less data is fetched. Vinted’s median PromQL query uses three matchers. Each matcher in the old format requires additional random index reads. With three matchers, Parquet is five times faster. Projections, which skip columns not needed for sum-by operations, add another 25% improvement on top of that.
Migrating Tenant by Tenant
The migration path runs without a cutover. A converter reads TSDB blocks from the original bucket after a two-day grace period, which gives the compactor time to finish vertical compaction. A new surf gateway reads from the Parquet bucket and serves queries. A single parameter scopes the converter to specific tenants using a matcher, so teams move one tenant at a time. Thanos store reads metadata files from the Parquet bucket, finds the source block IDs that have been converted, and drops those blocks from its own serving range automatically.
Q&A
What is the roadmap for merging Parquet into the main Thanos repo? Giedrius expects a migration to a shared Prometheus community library first, followed by architectural improvements to the compactor, then a move to the main repo in roughly half a year to a year. ▶ 22:13
Will the Parquet file format change in ways that require re-converting production data? Improvements like fast static string tables (FSST), which compress labels two to three times better than ZSTD, will arrive through the parquet-go library and should be transparent to users with no re-conversion needed. ▶ 22:48
Notable Quotes
in our case the median and average promill query has three matchers. Giedrius Statkevičius · ▶ 18:35
In this case it’s uh becomes five times faster. Uh so it’s quite a huge difference. Giedrius Statkevičius · ▶ 18:53
I would personally invest into parking Giedrius Statkevičius · ▶ 24:07
we would go from 700 terabytes of data to I think 400 terabytes of data. So that’s uh 300 terabytes less to store the same thing without any loss in precision Giedrius Statkevičius · ▶ 12:10
Key Takeaways
- Parquet cuts Vinted’s 700 TB metrics store by 300 TB at minimum, with no precision loss.
- Three-matcher queries run 5x faster under Parquet than under the current TSDB format.
- A tenant-scoped converter enables zero-downtime, incremental migration to Parquet today.
About the Speaker(s)
Giedrius Statkevičius is a site reliability engineer at Vinted, where he focuses on internal monitoring infrastructure. He has been a Thanos maintainer for more than five years.