AI
Manual Detection Creation Takes Hours
Raphael’s first task was improving GitHub coverage. He wanted to detect disabled branch protections. He read GitHub docs, checked actual logs, wrote an SQL query, built tests, and watched them fail. He reviewed code, re-read docs, and fixed the query. Tests passed. In production, the detection broke. He reviewed CloudWatch, logs, and code again. The whole loop took about 2 hours. The thinking part was 5 minutes. The rest was repetitive work.
Two Workflows Handle Creation and Maintenance
The system has two phases. For new detections, a workflow gathers context from a Notion log source wiki, S3 log samples, and Glue schemas. It asks the AI to name the detection, set severity, and choose scheduled or streaming. A coding agent in a Lambda runs linting, tests, and an Athena skill to validate SQL queries. For maintenance, a separate lambda bundles the detection repo into an ECR image. A Linear ticket triggers it to add users to allow lists or fix bugs.
Idea Generation Uses Vector DB and Public Research
For common SaaS apps like Okta and GitHub, the system feeds public research and blog posts to the AI to generate detection ideas. For internal tooling and IoT devices, a vector DB pulls the company risk registry, log inventory, and MITRE ATTACK strategies. When a new log source is onboarded, the AI searches the knowledge base and proposes alerts. The downstream workflow then generates detection code automatically.
Orchestration Through Linear Removes Local Dev
All information lives in Linear tickets. Adding a label triggers a webhook that routes to the correct workflow. Other teams contribute detection ideas without understanding the pipeline. They write a ticket, kick off the workflow, and the detection team only reviews the PR. No local development environment is needed. The system has created over 150 new detections. The bug-fix lambda runs every week.
Q&A
How long would it take to build this system from scratch? One engineer could finish the initial build within one quarter, but the team iterates by collecting feedback and replacing components as new tools emerge. ▶ 23:21
How do you validate detections are correct? If a log contains a known malicious event, verification is straightforward. For new detections, code review does not go away. The team is building an AI-based evaluation framework to measure if prompt changes improve or degrade results. ▶ 24:56
How do you handle poor documentation and unseen log fields? The team feeds logs through the testing pipeline. If a field only appears during a rare malicious event and documentation lacks it, it is hard for anyone to detect. ▶ 26:38
Do you build composite detections or just atomic signals? The AI workflow does not currently correlate multiple events. Alert enrichment uses Substation to add context like department and role, which helps triage. ▶ 27:52
Notable Quotes
i think first of all this is a very clearly defined problem so this journey that i went through in the beginning you literally go through every single time Raphael Ruban · ▶ 8:16
we realized that every time you do this workflow you need the same kind of information you need to know what log sources we have available what the log samples are and so on so we can just get it for the agent right away we don’t need to ask the agent to get it itself Raphael Ruban · ▶ 12:08
the only really manual work that’s left is you need to review these linear tickets if you agree with the detection and then review the pr at the end and you add a label it’ll go to a webhook and then get routed to our corresponding workflow and then you just have the pr at the end Raphael Ruban · ▶ 20:36
Key Takeaways
- Agentic workflows cut detection creation from 2 hours to 20 minutes.
- Manual context management beats letting the agent fetch its own data.
- Linear tickets as the source of truth enable other teams to contribute.
- A dedicated Athena skill fixes AI capitalization errors in SQL queries.
About the Speaker(s)
Raphael Ruban and Chen Cao are on the Detection and Response team at Verkada.