AI
Why Bootstrapping Deserves Its Own Discipline
Bootstrapping is when your fleet is most exposed. Anything slipped onto a machine before the baseline gets set may stay hidden for years. The process is also error-prone and left undocumented, it becomes “Joe’s problem” until Joe leaves. If you don’t plan the downtime to maintain the system,
“it will plan it for you.” — Michael Aldridge
Aldridge defines scope first: zero is machines racked with layer 2 up; production is real workloads running with no bootstrap toggles still active. Disaster recovery looks identical to day-zero bootstrapping, so a tested bootstrap doubles as a tested recovery plan.
Why Nomad Fits the Bootstrap Role
Nomad’s single statically linked Go binary has no external dependencies beyond OS primitives. At bootstrap time, when normal tooling isn’t installed yet, that matters. Aldridge measured 135 MB resident memory for the full bootstrap controller set on a test cluster. CPU load is bursty, not sustained.
Nomad is also passively stable. During a multi-day turnup, put it in place, walk away, and find it still running on return. For deployments where someone else executes the plan, possibly in a country where you’re not authorized to work, the process needs to be hand-off-ready.
The Bootstrap Job Stack
The bootstrap job listing walkthrough (11:12) shows a short Nomad job file that covers TLS certificates, a Traefik proxy, a local container registry, and terra-state (a minimal Terraform HTTP state backend). TLS comes first because bootstrapping is the ideal moment to compromise an environment. The registry and terra-state exist for the same reason: during a recent turnup, the site ran on a 1 Gbps AT&T line instead of the normal 400 Gbps backbone. Local caches mean you can start installs and go to dinner. The whole stack fits in 135 MB.
From Bare OS to Cluster Member
Once the bootstrap stack is up, dnsmasq sends a DHCP addendum and TFTP delivers the install image. Shoelaces automates image assignment so an IC can walk the aisles pushing power buttons without tracking which node gets which OS. Ansible pushes over SSH, installs Nomad, and joins each machine to the cluster.
The last Ansible step deletes the user that ran the playbook, clearing the final bootstrap toggle. Keeping the bootstrap compute attached after this gives you a lifeboat: if the main cluster fails, critical control-plane services keep running on hardware that depends on nothing else.
Ansible in Constant Time via Nomad System Batch
Ansible’s execution time scales linearly with host count and gets painful past about a thousand nodes. The Ansible-in-container bind-mount trick (18:24) solves this: run Ansible inside a container, bind-mount the host’s root filesystem to /host, chroot into it, and use the local connection plugin. Scheduled as a Nomad system batch job, the task fires on every node simultaneously, so Ansible runs in
“constant time regardless of fleet size.” — Michael Aldridge
Bootstrapping and maintenance collapse into one process:
“your bootstrap processes don’t go stale.” — Michael Aldridge
Cron-schedule the job for drift detection. With this approach you can stop sshd entirely.
Bootstrap Hardware: Mini PCs and the Switch as a Server
The bootstrap controller must fit in a carry-on. Aldridge used a GMK NUCBox5, a pocket-sized mini PC with 12 GB of RAM, the source of the 135 MB memory figure. Memory can be oversubscribed during bootstrap since no user traffic is at risk; if a task crashes, Nomad reschedules it.
Teams that want no dedicated hardware have another option. Arista exposes the supervisory controller on their high-port-speed switches. It runs a GPU-accelerated fork of enterprise Linux. That is enough compute to run the full bootstrap stack and bring a cold data center to production without any external box.
Notable Quotes
it will plan it for you. Michael Aldridge · ▶ 5:15
constant time regardless of fleet size. Michael Aldridge · ▶ 19:25
your bootstrap processes don’t go stale. Michael Aldridge · ▶ 20:13
Key Takeaways
- Bootstrapping and disaster recovery share the same process, so a working bootstrap is also a tested recovery plan.
- Nomad’s single binary uses 135 MB resident memory and runs the full bootstrap stack on a pocket-sized mini PC.
- Running Ansible as a Nomad system batch job makes fleet configuration run in constant time regardless of node count.