AI
Why LLMs Fail Without Context
Manu was updating a Terraform module for AWS Glue version 5.0 when he broke it. No tests existed. Writing them manually was slow, so he looked at generative AI. The immediate problem: any LLM is trained to a cutoff date and knows nothing about your codebase, your organization’s style, or how your team collaborates. Feed it a bare prompt and the output reflects that ignorance. The rest of the session is about closing that gap with structured context.
Automated Test Generation for Terraform Modules
One prompt asked Q Developer to create Terraform tests for the Glue module. The Q Developer generates 17 Terraform tests (02:31) started by inventorying the module’s resources, then detected Terraform 1.13 to confirm parallel execution was available. It proposed 17 tests, ran terraform test, and got 4 passing, 1 failing, and 12 skipped. The agent iterated until all 17 passed. Manu asked for documentation in the same session. The README listed every resource, the version constraints used, and flagged one test that would not pass, pointing back to a conditional in the module.
Context Engineering: Rules, Agents, and MCP
Manu’s prompt was backed by seven markdown files in a rules/ folder: testing strategy, mock provider patterns, parallel execution guidance, success criteria, and a troubleshooting reference. He built them by asking Q to reflect on past sessions and extract what worked. Global rules apply across all projects. Project-level rules live in the working directory. Agents in QCLI let you scope which MCP servers each task can see. A test agent and a scan agent can share similar context documents but pull different toolsets. On MCP: pick servers for the specific task and keep credentials out of plain-text config files.
Building a Terraform Action on Undocumented APIs
Terraform Actions were announced at the conference that day. No documentation existed. Dirk set out to build an SNS-publish action from a blank canvas using the Live Terraform Action build (93 minutes) (18:15) approach: a five-step plan first, then step-by-step execution. Q produced code in 6 minutes but then hit type mismatches, a missing test trigger, and an IAM policy without a version field that caused the test to silently hang for 5 minutes. Dirk also gave it bad information once and had to self-correct. One hour and 33 minutes of real time later, tests passed and the pull request was ready.
The Engineer Stays the Decision-Maker
After 93 minutes of back-and-forth, Dirk named what AI actually is in this workflow.
AI is not the hero. It’s the Chewbacca. — Dirk Avery
Manu added that working with Q made him a better communicator: he had to calibrate context precisely, giving enough to guide without overloading. The AI does what you ask, not necessarily what you mean. The engineer still catches its mistakes and redirects it when it goes wrong. Both closed with the same point: “AI might replace engineers that don’t” use it.
Notable Quotes
AI might replace engineers that don’t Dirk Avery · ▶ 29:54
AI is not the hero. It’s the Chewbacca. Dirk Avery · ▶ 28:58
which I love because Q is never tired Dirk Avery · ▶ 21:08
Key Takeaways
- Seven context files fed to Q Developer produced 17 passing Terraform tests from one prompt.
- Scoped agents control which MCP servers and tools are trusted, reducing noise and risk.
- After 93 minutes of real-time iteration, Q shipped a Terraform Action with code, tests, and docs.