AI
Why Production Visibility Is Hard
Half the audience raised hands for CSPM, but only one person was confident they knew real production state. Static posture tools show MFA is enabled but cannot confirm enforcement. A policy deprecated two years ago still allowed password-only authentication. Production environments are too complex for manual review. The gap between configuration and reality always exists. They found dormant admins and service accounts performing unexpected actions. Real telemetry is the only way to close the gap.
Building an Environmental Intelligence Report
Query logs where they live: Athena for S3, Splunk API for Splunk. Save queries in Jupyter notebooks. Use Claude code to analyze outputs. Categories include privileged actions, geographic locations, app usage, MFA enforcement, failed actions, and service accounts. Findings: dormant admins, an admin using a service account from a home network to escalate privileges, and a read-only service account performing write actions. The report is regenerated weekly to track changes.
Generating Consistent Reports with LLM Skills
Use a skill file with a template to ensure repeatable output. Sub-agents break the task: the longest report was 60 pages. A validation step checks each section and re-triggers if too short. Prompting tips: require citations, enforce hedging, separate fact from inference, challenge claims with benign explanations. The LLM must ask for missing data instead of inventing it. Consistency is critical for sharing across the organization. You own the report. You own the report, so review carefully.
Driving Action Across the Organization
Lead with impact: 300 over-privileged roles means what risk? Frame for management with weekly cadence to show progress. For technical teams, link to exact logs that prove the finding. Cost savings: six customers saved multi-million dollars in cloud spend by identifying unused resources. Track resolution together. Consistency builds trust and investment. The value compounds as you show month-over-month improvement. Unauthorized AI usage is a common blind spot found in every organization.
Q&A
Can you give an example of when LLM created a really bad report? The LLM once told a customer they had no MFA because it misread authentication data. ▶ 20:59
How do you convince customers it’s okay to use Claude to analyze data? They use Bedrock in a single-tenant VPC with a promise that data is not used for training. ▶ 22:06
Why do we need LLMs at all? Isn’t it just a scheduled report? LLMs connect dots across data sources and reason at scale, which is impossible with static reports. ▶ 23:09
How do you manage large logs within context windows? Use sub-agents to process specific outputs and a summarizing agent to combine them. ▶ 26:28
Notable Quotes
the gap between the static configuration and the policy to reality in production is something that always exists Shachar Hirshberg · ▶ 19:14
if you shared it internally, it’s now your mistake Hadar Waldman · ▶ 14:56
we process over a 10 petabytes per day Shachar Hirshberg · ▶ 27:43
unauthorized AI usage in organization. You always find things there, always. Shachar Hirshberg · ▶ 28:59
Key Takeaways
- Static posture tools cannot replace real telemetry analysis.
- LLM-generated reports need consistency through skills and sub-agents.
- Weekly cadence and linking to evidence drives action from leadership and technical teams.
About the Speaker(s)
Hadar Waldman leads security research and is part of the founding team of Artemis, the world’s first AI-Native detection and response platform. She previously led Check Point’s Incident Response Services, guiding organizations through high-stakes cyber crises.