Limits of Current AI Detection: Host-Based and Network-Based Coverage Gaps

▶ Watch (03:34)

Every AI security vendor today sells one of two things: a host-based solution using eBPF or ETW, or a network-based solution sitting in front of the API as a prompt firewall. Both categories work on the same narrow window — the moments when a prompt exists as plain text. Once the model starts processing tokens, they go dark. Regex, input sanitization, and LLM-as-judge checks all operate in that same plain-text gap, leaving the bulk of a model’s internal state unobserved and undetected.

Glass-Box Security: Intent and Strength via Mechanistic Interpretability

▶ Watch (05:23)

Glass-box security rests on two measurements: intent and strength. Intent is captured by hooking a model’s forward pass and collecting activation vectors, the high-dimensional representations the model actually thinks in, then running cosine similarity against a known concept vector. Cosine similarity tells you whether a prompt points in the direction of, say, illegality. Strength fills the gap: it measures how much of the activation’s total magnitude aligns with that direction, using scalar projection to separate a dominant concept from the superposition noise that compresses thousands of ideas into a finite latent space. Together, the two let a detection engineer reason about a space that humans cannot visualize, without touching a single token of plain text.

Forward-Pass Hooking and Latent Space Geometry

▶ Watch (08:37)

Hooking the forward pass requires no backward passes. Attach a hook to residual-stream layers, run inference, collect activation tensors. Cosine similarity tells you whether a prompt points toward a target concept. That is not enough. Superposition packs thousands of concepts into fewer dimensions, so directions bleed together. Scalar projection — the dot product of the activation against the concept direction — measures how much of the tensor’s magnitude belongs to the target concept. Layer selection is empirical: run prompts through the hooked model, find which layers activate most strongly, instrument only those. No universal threshold exists; each model needs its own calibration.

Behavior-Based Detection Manifolds and Detection Rules

▶ Watch (13:50)

Cosine similarity tells you a prompt points toward “illegal.” Scalar projection tells you how much activation mass that concept commands. Together they form a detection manifold, built from hooks across multiple model layers. A concrete rule: block any request with more than 85% cosine alignment to an illegal-intent vector where that concept also makes up a large share of total magnitude. Gemini shows why strength matters: it refuses “how do I rob a bank” but answers “how do I steal a pen” with a quip (13:20). Keywords miss that split. Manifolds catch it, and allow in-flight remediation rather than post-hoc logging.

Engineering Solutions for Activation Data, Scale, and Universality

▶ Watch (17:09)

Four barriers block this approach: frontier models don’t expose activations, activation data is enormous (GPT-OSS 20B produces 4 MB per token and 10 TB for a full context window), detection content is hard to write for non-AI engineers, and rules can’t be universal across every agent and use case.

The solutions are practical. For closed models, run a smaller instrumented “canary” model in-line or asynchronously. For data volume, hook only the residual stream, skipping the quadratic cost of self-attention heads, and monitor only the layers relevant to a given detection. For universality, Hurd argues for extending existing open-source formats: add AI-native modules to YARA and adopt Cedar rather than inventing new standards that detection engineers have to learn from scratch.

Agentic Threat Surface and Semantic Traceability

▶ Watch (19:26)

Agents operating autonomously will try packet-capture tools to execute binaries with elevated permissions, because that knowledge is in their training data. Syntactic rules cannot cover every variation. What’s needed is knowing which parts of the model activated and which were ignored — semantic traceability, not syntactic traceability. A YARA rule blocks any request where a file-deletion intent fires at specified layers above a set magnitude threshold (21:07).

“we have to intercept the thought before the action occurs” — Carl Hurd

“semantic observability is really the new foundation for security moving forward” — Carl Hurd

Q&A

LLMs are poor at calibrated confidence scores — how do relative intent/strength measures account for that, and can intent–strength pairs be stored as vectors on two axes? Hurd confirmed the two-axis vector framing is valid, and explained that intent/strength scores are relative to each specific model, not absolute, because intents are captured empirically by running thousands of prompts through that same model and comparing its activations against themselves. ▶ 22:51

Could intent-shift detection across a RAG pipeline serve as an indirect prompt-injection signal? Hurd confirmed it works, distinguishing context-enhancement RAG (where a single prefill-stage check suffices) from tool-based RAG (where before/after comparison is needed), and noted the same approach can universally block jailbreak concepts at the prefill stage. ▶ 24:51

What happens to detection manifolds when the model is fine-tuned or otherwise updated? Hurd said it depends on which layers are frozen: if fine-tuning modifies the layers where manifolds were built, you need to recapture empirical data or retune that portion of the detection. ▶ 27:23

Notable Quotes

we have to intercept the thought before the action occurs Carl Hurd · ▶ 21:48

semantic observability is really the new foundation for security moving forward Carl Hurd · ▶ 21:53

sovereign infrastructure is a requirement for anyone that wants to take secure model usage seriously Carl Hurd · ▶ 22:01

you can’t write syntactic rules for every possibility Carl Hurd · ▶ 20:22

Key Takeaways

  • Instrument model residual streams with forward-pass hooks to detect malicious intent before any action executes.
  • Combine cosine similarity for intent direction with scalar projection for concept strength to write threshold-based detection rules.
  • Deploy a canary model in-line when frontier-model activations are unavailable; own your inference stack for serious AI security.