What Cortex Does and How Its Architecture Scales

▶ Watch (1:49)

Cortex accepts metrics from Prometheus scrapers or OpenTelemetry collectors via remote write or OTLP, stores them in object storage like S3 or Azure Blob, and serves long-term queries through Grafana or similar tools. Every component, distributor, ingester, querier, is horizontally scalable. Ingesters hold time series in memory and flush two-hour blocks to object storage. A compactor merges those into larger blocks, cutting query overhead. A gossip ring protocol lets components reroute traffic when a node fails.

New Ingestion and Query Features in Cortex 1.20

▶ Watch (8:20)

Release 1.20, shipped in November, added experimental remote write 2.0 support with better retry handling. Ingestion now uses a stream-based push between distributors and ingesters instead of a unary call, keeping connections alive longer. Native histograms gained out-of-order ingestion support, per-tenant observability, and a new sample-rate limit so one tenant cannot flood others. Multi-tenant queries now accept regex patterns in the tenant header, replacing long pipe-delimited lists of tenant IDs.

Dynamic Query Splitting and Ingester Resource Limits

▶ Watch (11:04)

Before 1.20, query sharding was static. Long time-range queries produced too many small shards and hammered the backend. Dynamic splitting calculates shard size automatically based on range and vertical splits, producing fewer, larger shards. A blog post with a QR code covers the math. Separately, ingesters can now reject queries when CPU or memory crosses a threshold. In the demo, a 60% CPU limit triggers rejections, preventing ingesters from running out of memory under query pressure.

Parquet Storage: Removing the Store Gateway Stateful Set

▶ Watch (16:46)

Cortex 1.20 introduced a parquet converter that rewrites time-series blocks into column-oriented Apache Parquet format, then stores them beside the original blocks in object storage. Prometheus queries typically select only a few label columns, so column-oriented reads skip irrelevant data. Philip from Shopify tested this and found it resolved high-cardinality query problems. The 1.20 architecture dropped the stateful store gateway entirely. The 1.21 release candidate brings the store gateway back in a revised form because its per-block caching improved overall query performance.

What Is New in the 1.21 Release Candidate

▶ Watch (19:01)

The 1.21 RC, available now for testing, promotes several long-running features from experimental to production-ready: ruler API, alert manager API, tenant federation, and cache instance limits. A new querier metrics feature, contributed by Ben, exposes which metrics are actually being queried so operators can audit whether stored data is used. The overrides API lets tenants modify their own rate limits directly, removing the need to file a ticket and edit a config map. Hard ceilings on those limits remain enforced by the platform operator.

CNCF Graduation Progress

▶ Watch (23:34)

Cortex is approaching 10 years old and is working to leave CNCF incubation. Adding company names to the adopters file helps the graduation case. A third-party public security audit started recently and runs for the next three months. It is the first public review; earlier reviews were internal. Community members can join a meeting every four weeks, ask questions in the Slack channel, or file GitHub issues.

Q&A

How does Cortex differ from Thanos? Cortex focuses on multi-tenancy and a SaaS-style API; Thanos focuses on the sidecar model. Blocks are cross-compatible, and at least one maintainer works on both projects. ▶ 25:27

Is the parquet feature shared with Thanos? The core of the feature is shared; some parts differ between the two projects. ▶ 27:09

Notable Quotes

it’s not really clear right now if really makes sense to have two. Let me put it that way. Friedrich Gonzalez · ▶ 25:52

you can have emojis in the metric name and the metric label values Charlie Le · ▶ 14:30

we are not competitors. We are more like collaborators. Friedrich Gonzalez · ▶ 26:12

am I actually uh quering the metrics that I’m storing right? Why am I using all these data for Friedrich Gonzalez · ▶ 22:27

Key Takeaways

  • Dynamic query splitting in 1.20 replaces static sharding and reduces backend load for long time-range queries.
  • Parquet block storage lets queriers skip irrelevant label columns, cutting query cost for high-cardinality data.
  • The 1.21 overrides API lets individual tenants adjust their own ingestion limits without operator intervention.

About the Speaker(s)

Friedrich Gonzalez is a software engineer with 20 years of experience at Apple and an open source maintainer for Cortex.