AI
Why Agentic Systems Demand Execution-Layer Defense
Agents are no longer just enriched LLM calls. When an agent can autonomously call APIs or query databases, it becomes a privileged system executor — and the threat surface shifts from bad words to unauthorized system execution. At Salesforce, Rittinghouse and Huang defend 12,000 unique daily active agents across 55,000 customer organizations. Static signatures don’t scale. Valid behavior varies wildly between agents, so they built defenses at the execution layer, which sits outside but works alongside the reasoning layer these agents run on.
Two Threat Buckets and the Content-Moderation Gap
Salesforce’s threat model splits into two buckets: platform-targeted attacks (misconfigured permissions, flawed Apex skills exposed to the reasoning engine) and abuse of legitimate agency, where attackers use an agent’s built-in capabilities to perform actions that are valid but malicious in context. Content moderation handles the first layer — filtering toxicity and prompt injection at the reasoning layer — but it cannot see the execution layer. It captures malicious intent without the high-fidelity signal needed to block automatically. And even a prompt ruled safe can trigger unauthorized data access or privilege escalation that content filters never observe.
Three-Level Ensemble Anomaly Detection
Traditional anomaly detection runs at the user level. Rittinghouse’s insight was that agentic systems add two more axes: the agent itself and the organization deploying it. Stack all three and you have an ensemble model where each level cross-checks the others. Noise drops because a session has to look anomalous at multiple levels before it scores as a real alert. From 1.8 million daily prompts, that cross-checking is what made the difference between an alert queue no one can triage and 30 actionable signals.
Feature Engineering: Data Depth, Breadth, and Rarity
The team tracked data access in two dimensions: depth (what proportion of the database an agent touched) and breadth (call frequency). PII field flags and per-column access rates added a rarity dimension on top. Every day, the model rebuilt a rolling historical profile of all sessions, scored new sessions against standard deviations from that baseline, and surfaced outliers in near-real time.
A custom query-complexity calculator built early on failed. Agents construct the queries themselves, so query complexity measured agent behavior, not user intent.
“make sure that your features are measuring what you think they are measuring” — Matt Rittinghouse
Dropping the calculator and eliminating multi-table joins cut training time by two-thirds.
Results, Agentic Triage, and the Road to Auto-Containment
The behavioral anomaly detection model filtered 1.8 million daily prompts down to fewer than 30 alerts. Each alert fires as a structured JSON payload, ingested by a secondary agent that synthesizes a plain-English anomaly summary — no security background required. See a live alert payload handled by the triage agent. The current pilot accepts 12-to-24-hour detection latency running in batch jobs. The architecture is already built for the hot path: once baselines move to a high-speed cache, inference runs in flight. When a session crosses the statistical threshold — behavior that is essentially impossible to be normal — the system kills the session, revokes the token, or triggers a block immediately, without waiting for an analyst.
Four Lessons from Building in Production
Logs came first. Early on, engineering logs recorded what happened but not which agent or user caused it. Without structured events linking the invoking user ID to the agent ID, you cannot distinguish a power user from a compromised agent. Partner with engineering before writing a single detection rule. Second: profile normal now, because
“a signature-based approach is essentially just a blackbox failure” — Millie Huang
You need the statistical mean before deviation means anything. Third, their SQL complexity parser consumed significant time and then failed — agents constructed the queries, not attackers. Simple invocation counts and data-rarity features won. Fourth: expect noise for the first 14 days of a new agent’s life. Build that warm-up into your SOC playbook.
Q&A
How did you beat the curse of dimensionality and avoid over-engineering features? Matt ran multiple rounds of PCA-inspired feature analysis to identify which features contributed signal, culling the set down to the minimum with predictive power, while Millie added that they built this programmatically — treating threshold tuning as hyperparameter optimization to control detection volume. ▶ 17:13
Did you build confidence scoring on alerts, and is there an annotation feedback loop? Deviation-based scoring gives a confidence-interval-style measure of how far out of character a session is for each axis, though they acknowledged the scoring still needs tuning and annotation work with internal teams is ongoing. ▶ 18:36
What are SOC analysts and incident responders saying about the signals, and what actions do they want — especially around auto-containment? They chose a simple distance-based algorithm specifically to trade accuracy headroom for interpretability, then layered an LLM on top to produce human-readable summaries; auto-containment is not yet live but the pilot is releasing to customers this month, with a purple-team red-team exercise planned to tune the model’s blocking confidence. ▶ 19:33
Notable Quotes
a signature-based approach is essentially just a blackbox failure Millie Huang · ▶ 15:39
your funnel is only as good as your baseline Millie Huang · ▶ 16:37
simple telemetry like invocation count or sensitive asset frequency produced way higher fidelity signals Millie Huang · ▶ 16:25
make sure that your features are measuring what you think they are measuring Matt Rittinghouse · ▶ 11:38
Key Takeaways
- Instrument agentic logs as structured events linking user ID to agent ID before writing a single detection rule.
- Profile normal behavior starting today — you need the statistical mean before you can act on deviation.
- Simple invocation-count and data-rarity features beat complex query parsers; ship the simpler model and tune from there.