Why Go Misreads CPU Capacity in Containers

▶ Watch (0:55)

Goroutines, Go’s basic concurrency unit, start with a 2 KB stack and get multiplexed across OS threads. The number of those threads is controlled by GOMAXPROCS. Until Go 1.25, the runtime set GOMAXPROCS by calling runtime.NumCPU, which returns the logical CPUs visible on the host node, not the container’s allocated share. On a 24-vCPU Kubernetes node, a pod with a 2-CPU limit still caused Go to create 24 scheduler threads, far exceeding what the container’s cgroup quota would allow.

How CPU Limits Translate to Time Budgets

▶ Watch (2:15)

CPU limits in Kubernetes are enforced by the Linux CFS (Completely Fair Scheduler) using a 100-millisecond accounting window called the CFS period. A container limited to 400 millicores gets 40 milliseconds of CPU time per 100ms window. An API call that needs 200ms of CPU time will be throttled across multiple windows, taking up to 500ms to complete. Memory overuse triggers an OOM kill. CPU overuse just stalls the process until the next quota window opens.

The 24-Thread Problem on a 2-CPU Container

▶ Watch (3:57)

With GOMAXPROCS set to 24 on a node with 24 vCPUs, the Go runtime tries to run 24 OS threads simultaneously inside a container with a 2-CPU cgroup limit. All 24 threads compete for 40ms of CPU time per 100ms period. Throttling becomes severe and continuous. Kumar used this concrete example, a 24-vCPU node with a pod limited to 2 CPUs, to show that the mismatch between host visibility and container quota is the root cause of the problem.

Go 1.25: Reading Cgroups Directly

▶ Watch (4:55)

Go 1.25, released roughly six months before this talk, reads the cgroups bandwidth limit and sets GOMAXPROCS to match. It also polls that limit periodically, so a pod resized in place via Kubernetes’ in-place pod resize feature will update GOMAXPROCS without a restart. One caveat: fractional CPU limits cause Go 1.25 to ceil the value, so a 1.5-CPU limit sets GOMAXPROCS to 2, and throttling can still occur. Integer CPU limits are the safe path. Manually setting GOMAXPROCS disables all of this automatic behavior.

The Pre-1.25 Fix: Uber’s automaxprocs

▶ Watch (5:56)

Teams not yet able to upgrade to Go 1.25 can use automaxprocs, an open-source library from Uber. Adding the library and a blank import applies the same cgroups-aware GOMAXPROCS logic at startup. No other code changes are needed. Kumar recommended pairing either fix with a performance test to find the correct integer CPU limit for the workload rather than relying on fractional values that trigger the ceiling behavior.

Notable Quotes

But Go until 1.25 was not container aware. Adarsh K KUMAR · ▶ 01:05

Go max procs would get set to 24 because on the host I can see 24 nodes, which is what the runtime.NumCPU would return. Adarsh K KUMAR · ▶ 04:37

If you go above on memory, you get OOM killed, but CPU again it’s a time based thing, so you get throttled basically, right? Adarsh K KUMAR · ▶ 02:56

All of this will get disabled if Go max procs is manually set. Adarsh K KUMAR · ▶ 05:51

Key Takeaways

  • GOMAXPROCS defaults to host CPU count, not container CPU limit, until Go 1.25.
  • A 400-millicore limit gives only 40ms of CPU per 100ms window, causing visible latency spikes.
  • Go 1.25 reads cgroups bandwidth and re-polls it when Kubernetes resizes pods in place.
  • Fractional CPU limits cause Go 1.25 to ceil GOMAXPROCS, preserving some throttle risk.
  • Uber’s automaxprocs library delivers the same fix with a single import on older Go versions.

About the Speaker(s)

Adarsh K KUMAR is a Principal Engineer at Rapido with 13+ years of experience running infrastructure and distributed systems at scale.