Building a Self-Service Vault Platform at Scale

▶ Watch (01:00)

GoDaddy started with a community edition Vault cluster in 2013 and expanded it into an enterprise-wide secrets service running on OpenStack hypervisors inside on-prem data centers, deployed in Docker Swarm. Two authentication paths handle all access: OIDC through Okta for humans, using group claims mapped to Active Directory groups, and AppRole for machine accounts. A self-service portal lets developers request KV2 engines and join AD groups without opening a ticket. ServiceNow records feed into an intermediary JSON structure, which Terraform then reads to configure Vault automatically.

Automated Secret ID Rotation Without Application Redeployment

▶ Watch (04:59)

AppRole secret IDs at GoDaddy carry a 90-day TTL. A Python script enumerates every role’s secret ID accessors and checks whether a valid ID exists more than 45 days out. If not, it generates a new one and writes it to the KV2 engine at a fixed path (approle/{role-name}), where the application has read-only access. The app compares its cached ID against what it reads from that path, then swaps the value in memory or on disk. No redeployment. Jenkins runs the rotation loop and routes failure alerts to Slack.

Seven Terraform Modules and the Data Structure Rules Behind Them

▶ Watch (08:16)

GoDaddy wrote seven Terraform modules covering every Vault resource type: cluster provisioning on OpenStack, static config, AppRole auth, KV2 engines, LDAP app roles, and AD account bindings. With roughly 450 LDAP auth mounts in production, one module fix propagates everywhere at once.

“everybody should be using modules.” — Jeremy Darr

The data structure feeding those modules must stay shallow. A relational multi-structure approach broke Terraform’s iteration. The working pattern: one JSON file, lists of dictionaries, sliced in locals. State files live in on-prem Ceph object storage. Aborting a Terraform run mid-flight corrupted a dev cluster’s state file and forced a full rebuild.

Anti-Patterns, State File Pitfalls, and Why Planning Beats Refactoring

▶ Watch (19:11)

Adding an environment label to AppRole names after launch meant a wave of move.tf blocks and path migrations that broke users already relying on the old convention. Path changes do the same damage.

“users set the pattern.” — Jeremy Darr

Darr involves power users in beta before launch. When he built in isolation, adoption stalled. He also recommends switching from the deprecated -state flag to Terraform backends, and keeping Vault policies scoped to exactly what each machine account needs, nothing more.

Notable Quotes

everybody should be using modules. Jeremy Darr · ▶ 8:19

itself, you want to keep that shallow. Jeremy Darr · ▶ 16:04

users set the pattern. Jeremy Darr · ▶ 22:35

Key Takeaways

  • Automate AppRole secret ID rotation at half-lifetime to avoid application disruptions during credential expiry.
  • Use a single shallow data structure of lists-of-dictionaries to feed all Terraform modules; avoid relational patterns.
  • Plan naming conventions and path structures before launch — post-launch changes break users and require painful state moves.