AI
The Runner Gap at Mercedes-Benz
Mercedes-Benz runs GitHub Enterprise internally but the license didn’t include hosted runners. Teams built their own ad-hoc setups, which created duplicate effort and security gaps. The platform team had no budget or mandate to fix it. Their response: a one-day design doc stating what to build, why, and how. That document became both a decision-making reference and a pitch to management. Every time a design choice came up, they checked it against the doc to stay aligned with the original goal.
How GARM Was Built for Flatcar Linux
Gabriel built GARM in 2022 to move the Flatcar Linux CI off an internal Jenkins instance. GitHub’s hosted runners had a 6-hour job limit and the Flatcar SDK needed actual VMs (device mapper, LVM) instead of containers. He found Arc but it only provided container runtimes. So he built GARM: a webhook consumer that spins up VMs, joins them to GitHub, and runs the job. The plugin interface works like Kubernetes CNI: an executable called with environment variables. This design made it possible to add AWS, Azure, Incus, and Kubernetes providers without changing GARM’s core.
Getting First Users: Labels and Friction Removal
Getting the first users onto Road-Runner was the hardest part. Mercedes used two tactics. First, spread the word on every messaging channel and in every competence group. Second, remove friction: new users need an immediate “aha moment,” not a configuration manual. The team noticed other teams already used the self-hosted GitHub Actions label. Road-Runner registered under the same label. When those teams enrolled, their workflows ran on Road-Runner automatically with no changes needed. They also added transparent HTTP proxy routing and remote shell access for debugging live VMs.
First Incident and the Postmortem Practice
The first major incident: the infrastructure layer failed to clean up consumed runners. Pools filled. No new runners could spin up. No workflow ran. Once the root cause was clear, the fix was fast. The bigger decision was what to do next. The team wrote a postmortem: timeline, root cause, and preventive measures. Users thanked them for the transparency, even after the failure. That cycle, writing a postmortem after every outage, became a standing practice.
12 Million Runs and Giving Back to Open Source
Four months to 100,000 runs. One year to 1 million. Today: 12 million total and roughly 700,000 per month. Runner startup sits at 3 to 9 seconds, even on ARM64. The Kubernetes operator that manages GARM in a cluster came from the Mercedes team and was open-sourced. Contributing back also meant reporting edge cases that only appear at scale. Running GARM at 700,000 monthly jobs surfaces failure modes the original developer would never see on a small cluster.
Notable Quotes
written in Go, the best language on the planet. Michael Kuhnt · ▶ 5:26
And users need this aha moment Michael Kuhnt · ▶ 13:15
people don’t expect perfection Michael Kuhnt · ▶ 17:58
Key Takeaways
- Write a one-day design doc before building anything: it anchors decisions and serves as a pitch.
- Test an open source project’s collaboration style with a small pull request before committing.
- New users need an immediate aha moment; use the self-hosted label to avoid forcing workflow changes.
- Write a postmortem after every incident: transparency builds trust more than uptime perfection.
- At scale, give back edge cases and tooling to the upstream open source project.
About the Speaker(s)
Gabriel Adrian Samfira is an open source engineer and maintainer of Flatcar Linux, with nearly two decades of experience in cloud infrastructure. He built GARM to solve a real CI problem in the Flatcar project and contributes to Coriolis and powershell-yaml at Cloudbase Solutions.
Michael Kuhnt is a software engineer at Mercedes-Benz Tech Innovation, focused on platform engineering and building internal developer tools. He helped scale Road-Runner from a zero-budget initiative to a platform handling 700,000 workflow runs per month.