AI
The Scalability Gap in Security Guidance
At most organizations, security engineers are outnumbered. The ratio is low enough that one-on-one guidance doesn’t scale. Developers ask the same questions across Jira tickets, Slack, threat modeling sessions, and design reviews — and each answer depends on which security person has bandwidth that day. Meanwhile, documentation written by security champions and SMEs sits in wikis, reaching the wrong person at the wrong time, or not at all.
Why Generic AI Fails and What the System Needs Instead
Ask a generic LLM “what authentication standards should I follow?” and you get generic answers. It knows nothing about your company’s approved libraries, internal APIs, or platform-specific policies. Worse, when the context window fills with organizational docs, hallucination risk climbs.
The solution Mukherjee and Datta Gupta landed on: a vetted, org-specific guidance store that feeds every platform. One central source, consistent output, no hallucinated policy. The system had to work the same whether a developer asked from Jira, Slack, or their IDE.
Automated Document Ingestion Pipeline
The ingestion pipeline starts with a Git repo. Security champions submit a metadata file containing the URL of the document to ingest. A PR review validates it, the merge triggers a pub-sub downloader service that fetches the content and hands it to an ingestor, which generates embeddings and writes them to the vector store. A cron job re-checks every previously ingested URL for content changes and re-ingests when it finds a diff. Once ingestion completes, a Slack notification fires and an eval workflow kicks off, testing the new content against a reference question-and-answer set to verify correctness and relevancy before the document goes live.
Core Architecture: One Guidance Service Across the Entire SDLC
Every entry point, a Jira vulnerability ticket, a threat modeling platform request, or a plain chatbot question, feeds the same AI orchestrator. Before hitting the vector store, the orchestrator adjusts the system prompt for each input type so the returned guidance fits the context. The LLM then formats the response as a structured JSON payload configured per workflow. All requests log to LangSmith for traceability, and several workflows run online evals in the same pipeline.
Live Use Cases: Chatbot, Threat Remediation, and IDE MCP Server
Three use cases draw from the same vector store. A security chatbot answers policy questions like “How do I reach the security champion for product Y?” The threat modeling engine identifies threats, calls the service, and returns Adobe-specific mitigations. For Jira vulnerabilities, the service gives both a short-term fix and a long-term, class-level remediation for teams hitting the same bug pattern repeatedly. The furthest-left integration is the MCP server: a developer asks “How do I fix SQL injection?” in Cursor, and the MCP Server Demo in Cursor IDE (14:24) shows Adobe-specific guidance printed inline. The Slack Workflow Demo (16:00) returns the same output, confirming consistent guidance wherever in the SDLC the developer sits.
Challenges, Lessons Learned, and Key Takeaways
Evals were the hardest part. Building a golden dataset is slow and manual, but it’s what separates a demo from a production system.
“Evals is what will turn your AI experiments into production systems.” — Shruti Datta Gupta
Doc freshness runs a close second. Stale guidance in the vector store means bad recommendations, so the team runs a shared-responsibility model: evals flag outdated docs and owners push updates when content changes. Peer review in the ingestion pipeline is a deliberate bottleneck — it slows ingest but protects quality. On the tooling side, V1 required manual chunking and vectorization because LLMs couldn’t handle multimodal input. That work is now handled by off-the-shelf vendors. Preliminary testing of the security rules and skills files reduced vulnerabilities in generated code by roughly 70% (22:37).
Q&A
Can you describe the threat models you built and distributed through this tool? A separate team built the threat modeling engine, which parses wiki pages in a specific format to generate threats; Adobe’s guidance service then plugs in at that point to append org-specific remediation recommendations. ▶ 20:00
How do you enforce use of the MCP server and ensure consistent configuration across all developers? A Cursor extension ships all MCP configs and Cursor rules by default on sign-in, and developer adoption has been organic because engineering teams find the guidance useful enough to request it themselves. ▶ 20:47
Have you tried injecting these as agent rules or .md files or treating them as skills, and how did that go? Yes — they’ve added security rules as .md files and tested skills, with preliminary results showing roughly 70% reduction in vulnerabilities in generated code. ▶ 22:05
Notable Quotes
Evals is what will turn your AI experiments into production systems. Shruti Datta Gupta · ▶ 19:24
AI is useful. AI boosts productivity. But context is key. Shruti Datta Gupta · ▶ 18:50
When you make security zero calorie and seamless for developers, they will get on board and they will want to do the right thing. Shruti Datta Gupta · ▶ 21:40
Key Takeaways
- Build a single vetted vector store and route every security touchpoint through it instead of creating per-platform RAGs.
- Invest in evals before going to production — a golden dataset is the non-negotiable foundation for reliable AI guidance.
- Make security invisible in the developer workflow: org-specific MCP servers and IDE extensions drive adoption without enforcement.