Build your own Terraform provider

by Robin Norwood

Day 2 · Storehouse, Building D, Fort Mason · Hands-on Labs · 64 min · 4 min read

Infrastructure as Code

TL;DR

Robin Norwood walked attendees through the Terraform plugin framework by adding a resource and data source to an existing HashiCups provider. The lab covers two build scenarios — no provider exists, or the existing one won't work for your environment — then implements the full gRPC handshake between Terraform core and a custom provider written in Go. Key mechanics: schema definitions that map HCL to client library types, a configure() function that initializes the API client once and shares it across all resources, and a terraform.rc dev override that routes Terraform to a local binary during development.

When to Build or Fork a Provider

▶ Watch (05:54)

Over 400 official and 5,000 community providers exist in the Terraform Registry. Two reasons justify building or forking one anyway. First: the provider doesn’t exist. Internal tools at large companies often have APIs with no published provider. Second: an existing provider won’t work for your environment. Norwood’s example is a healthcare insurance company running VMware. Their security protocols required configuration the official VMware provider didn’t support, so they forked it, modified the authentication, and used the fork instead.

The second case is also common when a cloud vendor releases a new API before the Terraform provider catches up. You add the resource yourself and ship immediately.

How Terraform Core and Providers Communicate

▶ Watch (10:00)

Terraform core does one thing: parse HCL, build a dependency graph, and determine what needs to be created, read, updated, or deleted. It contains no AWS code, no Azure code, no Kubernetes code. Providers handle all of that. Core acts as a gRPC client and passes parsed configuration to providers, which run as gRPC servers and execute domain-specific logic. The provider itself should not call the target API directly. It translates the resource schema into a format the client library understands, then hands off to that library. Terraform core is only parsing; the provider does all actual logic work.

Provider Client Setup and Authentication

▶ Watch (21:42)

The configure() function runs before any resource or data source code. It reads host, username, and password from HCL or environment variables — environment variables are preferred because hard-coding credentials into a Terraform config is a mistake. After validation, it creates an API client and attaches it to the response object so every resource and data source in the provider can retrieve it later. The scaffold repo ships with this authentication flow already implemented. The lab step is adding the schema fields that map provider-level HCL attributes to the client library’s expected inputs.

Defining a Resource Schema

▶ Watch (30:11)

The schema is the contract between an HCL configuration block and the API client. Every field the API returns or accepts must be declared in the schema with its type and whether it is required (user-provided) or computed (API-returned). For the HashiCups order resource, id and last_updated are computed. Items are required. Each coffee object inside items has its own nested attributes — id (integer, required), name, teaser, description, price, image. If you skip a field the API returns, Terraform won’t surface it to users. If you declare a required field the API doesn’t accept, every apply fails.

Implementing CRUD and Testing Locally

▶ Watch (35:28)

Create, read, delete, and update follow the same six-step pattern: check that the API client is configured, retrieve values from plan or state, build the API request body, call the client library, map the response back into the schema, set state. One rule distinguishes create from everything else:

“need to set a plan ID. If you do not set” an ID, Terraform will not save the state file. — Robin Norwood

He walked through the terraform apply live demo — order created and verified in state (47:44) live, showing order ID 2 appear in state and matching it against a curl response. A terraform.rc dev override file points Terraform at the local Go binary instead of the registry, so changes take effect after go install rather than a full registry publish.

Q&A

Can you use gRPC directly instead of Go to write a provider? Theoretically yes, but HashiCorp has no client library for any language other than Go — if you build one in Python or another language, open source it. ▶ 17:31

Will there be an ‘actions’ function alongside resources and data sources in the plugin framework? An actions function is coming to the plugin framework in the next few months, at which point it will appear in the same location as resources and data sources. ▶ 28:23

Notable Quotes

And this is by and far the most common Robin Norwood · ▶ 7:40

need to set a plan ID. If you do not set Robin Norwood · ▶ 39:42

That’s how you know this is a live lab Robin Norwood · ▶ 1:02:12

Key Takeaways

Terraform providers are gRPC servers: core builds the graph, providers handle all API-specific logic.
Every resource’s create function must set an ID, or Terraform silently drops the state entry.
Use terraform.rc dev overrides to point Terraform at a local binary instead of the registry during development.

Filed under

Infrastructure as Code