AI
The Architecture: Neural Networks, MCP, and a Doom Deathmatch
Each Doom player runs as a PyTorch-trained neural network inside a Kubernetes pod. Up to 16 networks run per pod, each saving its best hyperparameters to an IPFS network shared across all pods. When one player wins, those weights feed back into an adversarial network so every subsequent player benefits. A ZDoom host pod coordinates the deathmatch, and Kubernetes DNS handles pod discovery. MCP servers inside each player pod scrape per-player metrics. A supervisor pod aggregates those metrics across all 16 players.
Model Training: Five Techniques, 14 Iterations
The team settled on five model components after trying many combinations. IMPALA handled distributed learning at scale. ResNet improved visual perception. LSTM provided memory and temporal context. NoisyNet pushed players to try new behaviors. Dueling heads improved action selection. Despite that stack, players spent the first 14 iterations punching walls. Pod restart times were also too slow with standard Docker containers, so the team switched to WebAssembly to get rapid restarts without reloading the full neural network each time a player was killed.
Observability Stack: OpenTelemetry, Hubble, and Tetragon
Everything is instrumented with OpenTelemetry, following the generative AI semantic conventions from the OpenTelemetry project. Cilium serves as the CNI, with Hubble visualizing pod-to-pod and cluster-to-cluster traffic flows. Tetragon, running eBPF underneath, provides process-level visibility showing exactly which system calls each pod executes. The Hubble UI renders a service mesh view across three clusters simultaneously. Grafana dashboards surface frag counts, MCP call latency per tool, span stability, top-scoring players at any moment, and the number of souls killed.
The Chaos Engine: Frag-Driven Pod Restarts
The chaos engine sits on top of the existing MCP logic. When a player gets fragged, the chaos engine reads that frag metric and restarts that player’s WebAssembly soul container. Only the soul restarts, not the full neural network, keeping restarts fast. A randomizer also changes the neural network temperature at each game restart, so players cycle between aggressive and cautious behavior. Without that temperature change, every game would produce identical flat-line statistics. The result is varied, observable gameplay that mirrors how real agentic workloads behave under failure conditions.
Personas, Trash Talk, and MCP Communication
Each player carries a persona, friendly, forgiving, aggressive, or otherwise, and all players communicate over MCP. Every frag translates into persona-driven messages between players. The supervisor agent calls GPT Mini to generate the actual trash talk text. Distributed traces in the observability dashboard show the full conversation chain: MCP supervisor speaks to individual players named with NATO alphabet identifiers (alpha, delta, charlie), and the results appear as banter visible in the trace spans. The dashboard filters let you track which player holds the highest frag count at any point.
Key Takeaways
- Training AI players to behave as intended took 14 iterations and multiple model components running together.
- WebAssembly soul containers let the chaos engine restart killed players without reloading the full neural network.
- OpenTelemetry semantic conventions, Hubble, and Tetragon together give pod-level, process-level, and network-level visibility simultaneously.
Q&A
Were the speakers tempted to play against the AI themselves? The goal was a fully automated system representing how agentic applications run today, though playing against the AIs remains possible. ▶ 26:02
How large was the team and how long did it take? Two people built it, leaning heavily on generative AI coding tools, with training running for days on a Mac before producing usable results. ▶ 27:12
Could a co-op team play against enemies instead of deathmatch? Yes, Doom supports co-op mode, and the setup maps directly to red-team versus blue-team security scenarios. ▶ 29:44
Notable Quotes
we end up building essentially a neural network per pod that essentially is being used per player to play Doom. Josh Halley · ▶ 05:19
we had 14 iterations of the player sucking before we got to a point where things kind of improved. Josh Halley · ▶ 15:22
every time that the player gets fragged, then it soul gets restarted. So, you’re not restarting a a large neural network every single time a player gets killed. Josh Halley · ▶ 18:18
I I don’t have that much trash talk in me. So we went for an outside source. Josh Halley · ▶ 29:20
About the Speaker(s)
Ricardo Aravena is a Cloud Native AI Lead and open-source contributor serving on the CNCF TOC. He co-chaired CNCF TAG-Runtime and led the publication of the Cloud Native AI Whitepaper, bringing more than 25 years of infrastructure and software engineering experience to the role.
Josh Halley is a Principal Architect in the CTO office at Cisco, where he focuses on emergent technologies and agentic AI systems. He co-organizes the CNCF Technical Community Group for Artificial Intelligence and is a published technical author.