AI
Why a Custom Operator Beat Manual Helm Deploys
b’nerd’s client wanted to resell a third-party application as a SaaS product across three tiers: basic, premium, and ultimate. He expected roughly 100 concurrent instances. Before this project, b’nerd deployed the same app by hand using the official Helm chart. That worked for occasional deployments. It does not work at scale, especially when each instance requires manual configuration steps at the end. The operator replaced all of that with a single API call from the client’s onboarding platform.
Four CRDs and the Logic Behind Splitting Them
The operator manages four custom resource definitions rather than one. ClientApp covers dependencies: Redis, a database, persistent volumes, and ingress. ClientAppProfile is cluster-scoped and defines which sub-applications are enabled per tier. ClientAppPool holds pre-warmed instances. A fourth CRD tracks external backend registration. Splitting the logic this way kept each reconciliation function focused. Merging everything into one CRD would have produced a reconcile function spanning thousands of lines and made debugging nearly impossible.
Pre-Warmed Pools to Hide Startup Latency
The basic profile takes about one minute to start. Premium and ultimate take up to three minutes. Waiting that long after signup is not acceptable. The solution is three pool CRDs, one per tier, each holding five pre-warmed instances at all times. When a user picks a plan, the operator pulls one instance from the matching pool, applies minor host-name and credential changes, and immediately replaces it with a fresh instance. The pool stays at the configured replica count continuously.
Connecting to External VMs Without Overreaching
Two backend services and a TURN server run on separate VMs and cannot move to Kubernetes. They share credentials and hostnames with every cluster instance. Rather than having the operator provision VMs directly, b’nerd placed a thin control API in front of each VM. The operator posts registration and deregistration calls to that API as instances are created or deleted. The operator stays responsible only for cluster resources, and the VMs stay responsible for their own state.
DNS, TLS, and Reusing What Already Exists
DNS and TLS looked complicated before implementation. External-dns watches ingress objects and creates DNS records at the provider automatically when a new hostname appears. cert-manager handles the ACME challenge and issues the certificate. The operator only needed to add the cert-manager annotation when it creates an ingress. No custom DNS logic was written. The same principle applied to the database: a community Postgres operator handles provisioning, and the operator waits for its secret before proceeding.
Testing, Debugging, and Deletion Order
Mock APIs are too limited for operator testing. A real dev cluster is required because unexpected behavior consistently appears only against a live Kubernetes API. Deletion turned out harder than creation: race conditions appear in teardown just as they do during install. One concrete example is deleting an S3 bucket after the credentials are already gone. Finalizers solve ordering problems. Detailed status updates and Kubernetes events in the reconcile loop are the main debugging tools when an instance stalls mid-creation.
Q&A
How did b’nerd deliver the operator and CRDs to the cluster? Flux manages a deployment manifest for both the operator and the CRDs, which avoids the CRD upgrade problems that Helm has. ▶ 29:51
Have CRD upgrades caused problems with Flux so far? No issues yet, though the operator has only been in production for a few months. ▶ 30:08
Notable Quotes
we are not talking about one or two instances per day but rather hundred maybe so a lot right Verena Traub · ▶ 02:23
you can’t delete an S3 bucket after you deleted the credentials. Haha surprise right? Verena Traub · ▶ 25:51
you start with a few hundred lines of code and then boop thousands line of code. Verena Traub · ▶ 24:38
the best case to test and try out and play around with operators is please use a real cluster. Verena Traub · ▶ 26:37
Key Takeaways
- Split operator logic across multiple CRDs early; merging them later costs far more time.
- Pre-warmed instance pools absorb 1-3 minute startup times before users ever notice.
- Reuse external-dns, cert-manager, and community operators instead of rebuilding their functionality.
About the Speaker(s)
Verena Traub is a Cloud Consultant and DevOps Engineer at b’nerd. After several years as an HR expert in IT, she switched to programming in 2020 and has focused on infrastructure and modern software development since then.