AI
The Cost of Falling Behind on Dependencies
Netflix runs more than 3,000 JVM applications across 5,000 repos, with 12,000 libraries and over 1,000 Java engineers. The JDK moved from Java 8 to Java 25 in a few years. Maven Central sees millions of new versions published continuously. Keeping up manually is not realistic.
“because really delivering code is hard.” — Roberto Perez Alcolea
Dependency conflicts stall deployments and debug cycles grow longer. Engineers spend time on version maintenance instead of product features. PRs get opened, sit unreviewed, and changes never reach production.
Build Opinions and Dependency Management
Netflix wraps Gradle in a custom distribution called Nebula. Every project gets build-file linting, dependency locking, and resolution rules. Substitution forces any version in a broken range to a known-good version. Replacement removes classpath collisions when two artifacts share the same classes. Alignment keeps co-released dependencies on the same version. All rules ship in a versioned jar that Nebula applies at build time across all 5,000 repos.
Before shipping a new rule set, Netflix runs an acceptance test: configure every active repo, build it, apply the update, build again. A high pass rate is the gate.
Artifact Observability with Astrid
Astrid indexes every Netflix deployment. When a new AMI or Docker image is published, it expands the manifest, finds the application jars, and builds a dependency graph stored in JVM memory via Hollow. Any engineer can then query which services run which library version.
The numbers Astrid surfaces are stark: only 6% of the fleet was on a given Guava version, meaning 93% needed to close the gap. New Spring Boot patch releases now reach close to 2,000 applications within 3 days. And 1.4% of libraries have more than 100 versions running simultaneously in production.
Distributed Refactoring at Scale
Netflix delivers changes across its repos in four modes. Nightly updates run via RocketCI on Jenkins: every repo gets cloned, dependency locks get updated, a PR is opened. If tests pass, it auto-merges. See the nightly update PR flow (22:05) for the engineer-side view.
For larger code changes, a campaign tool uses Astrid data to target which apps are still on Java 8 or Java 11, then runs a Docker container against each repo to apply the transformation and open a PR. Engineers can attach canaries as extra validation gates before merge.
“Merging pull requests is not enough.” — Roberto Perez Alcolea
Deployment Confidence and Continuous Delivery
Even with automated PRs, more than 50% of Netflix deployments were ad hoc or monthly. Spinnaker pipelines existed but went unused. Engineers did not trust automated changes from platform teams enough to let them merge.
Netflix surveyed developers and assembled a cross-functional team to identify gaps in build reliability and delivery infrastructure. They addressed those gaps, then ran a campaign to push teams toward continuous delivery. Daily deployments climbed above 60% of the fleet. Weekly deployments account for 26%. Confidence in platform-managed PRs improved.
Testing Infrastructure and Validation
Netflix built testing scorecards showing test success rate on mainline, execution time, and code coverage for every repo and org. Coverage earns a gold, silver, bronze, or stone rating. Teams at stone level get nudged.
“0% is not good there.” — Aubrey Chipman
Remote test execution via Dev Velocity distributes suites across agents so local and CI results stay consistent. Predictive test selection runs on PR builds, picking tests tied to changed code and previously flaky tests, to cut feedback time. Flaky tests get up to three auto-retries, with failure data surfaced in Dev Velocity.
Notable Quotes
because really delivering code is hard. Roberto Perez Alcolea · ▶ 4:59
Merging pull requests is not enough. Roberto Perez Alcolea · ▶ 25:14
0% is not good there. Aubrey Chipman · ▶ 32:00
Key Takeaways
- Nightly automated PRs keep 3,000+ Netflix JVM apps current without developer intervention.
- Astrid indexes every deployment’s dependency graph so Netflix can answer who runs what version in real time.
- Shifting from 50% ad hoc deployments to 60% daily required culture change, not just tooling.