AI
Why Detection Intent Needs Its Own Layer
Detection is the last line of defense after prevention fails. At Google scale, thousands of detections are created and updated constantly. Large global teams read, modify, and triage each detection. Knowledge cannot live in one person’s head. Bespoke implementations for each detection work to get off the ground, but long-term they add complexity. An abstraction layer takes more time upfront but reduces organizational complexity as the detection count grows.
The Six Detection Patterns
Singh and Amiri identified a minimal set of patterns from real detection logic. Simple patterns include predicates for filtering events, enrichments for adding context like VirusTotal hash lookups, and user-defined functions as an escape hatch for stateless logic. Complex patterns handle state: correlations link events near each other in time, thresholds fire when a count exceeds a limit, baselines compute fleet-wide rarity, and clustering groups repeated alerts to manage investigator time.
Expressing Intent in Go and Protobuf
Users write detection intent in a Go API that mirrors the patterns. The system translates that into a protobuf representation — a structured, implementation-agnostic form. This forces the separation between what the detection does and how it runs. A malware detection reads executions, enriches with VirusTotal, filters on malware verdict, and formats an alert. A brute-force detection reads logins, splits by success, applies a threshold, and correlates for missing successful logins.
Shielding Users from Time and Aggregation Complexity
Legacy systems forced users to handle event time versus processing time, late-arriving data, and deduplication. The new abstraction lets users express logic in event time only. Aggregation complexity is also hidden. Non-monotonic aggregations like counting login attempts per hour require careful latency balancing. Monotonic aggregations like threshold checks can return results immediately. A single correct implementation of late-data handling now applies across all detections, replacing bespoke code that previously caused false negatives.
Balancing Abstraction: Opinionated Where It Matters
The team applied a principle: be opinionated about hard problems, flexible about easy ones. The Go API is declarative and opinionated for pattern matching — correct data processing and joins are hard. User-defined functions in Go handle simple stateless transformations like mapping data structures. UDFs cannot access state or cause side effects. This balance lets security engineers write simpler, safer detection logic while platform teams optimize the underlying execution without changing detection intent.
Q&A
How do you handle alert cooldowns and stampeding alerts? That was outside the scope of this talk, but additional tools exist to manage investigator time by detecting and grouping similar alerts. ▶ 34:51
Does separating intent allow applying detections to multiple data sets? Yes. The same pattern can run against unified data or be copied across different data types. Both approaches are supported. ▶ 40:16
How have you thought about augmenting this with AI? Structured data and structured logic make automation work better. Many people are looking into this, and the system is designed to complement those efforts. ▶ 41:43
Were any primitives initially not seen as needed but became favorites? Enrichment was underestimated early on. The team ended up investing more time optimizing it as users demanded additional context in detections. ▶ 43:27
Notable Quotes
detection is the last line of defense. It’s trying to detect bad events that happened that you didn’t prevent. Gaurav Singh · ▶ 2:34
if you have a bunch of different detections and you have a bespoke implementation for each of them. This is a good way to get off the ground. Long-term this means that it more complexity for an organization Gaurav Singh · ▶ 6:10
security engineers will spend much more of their time reading, understanding, and maintaining existing detection logic. They’ll spend much more time doing that than writing new detection logic. Dario Amiri · ▶ 29:49
Key Takeaways
- Six detection patterns cover most real-world detection logic at Google.
- Separating intent from implementation lets users focus on business logic.
- Higher-level abstractions reduce cognitive load and enable system-wide optimizations.
About the Speaker(s)
Dario Amiri is a Senior Staff Software Engineer at Google, currently working as a tech lead on the Sign-in team in the Core Identity Platforms organization. Previously, he worked on the Detection and Response Platform team that builds detection systems at Google for seven years.