AI
Replacing Reliable Controls with Probabilistic AI
Niels Tanis starts with a thought experiment. Tell an engineer their firewall works 9 out of 10 times. Or authentication is 99% reliable. That might pass once. A year later, someone new will ask who accepted that design. Reliability is foundational to engineering. Tanis sees security teams letting AI impact the reliability of otherwise dependable controls. The root cause is the non-deterministic nature of LLMs and AI tools replacing deterministic security controls.
Speed Comes at the Cost of Missed Corner Cases
Tanis uses LLMs for two purposes. First, generating scripts that produce deterministic outputs every time. Second, combining non-deterministic summaries with deterministic detection rules. The problem emerges with speed. Asking an LLM for a script without deeply thinking through the problem space leads to missed corner cases. Tanis emphasizes that speed comes at a cost. The engineer skips interiorizing the subject, and bugs slip through.
AI-Generated Documentation Lacks Expert Context
Tanis’s biggest pet peeve is using AI to generate documentation. Good documentation reflects someone’s effort to understand the problem, think through trade-offs, and reach a conclusion. AI content creation breaks that link. Anyone, with or without expertise, can produce a 10-page report on a subject they spent zero hours studying. Readers cannot distinguish expert insight from AI-generated text. The relevance of the answer suffers.
Compounding Uncertainty Across AI Layers
Combining multiple AI factors creates a compounding effect of uncertainty. Tanis gives an example. Using a product with AI, reading documentation also written by AI, and summarizing that documentation with another AI client. Small mistakes, overstatements, and misunderstandings in each layer build up. The result is a theoretical solution handed to engineers who must figure out it is not feasible. Engineers then explain why X, Y, and Z make it unreliable.
Three Rules for Staying Grounded
Tanis shares three coping mechanisms. First, refuse to read documentation that appears AI-generated. Second, always validate claims from others who may not follow rule one. Technical verification is necessary. Third, combine deterministic and non-deterministic outputs in workflows. He acknowledges breaking his own rules sometimes. The key is accepting the consequences. Security engineers must know when reliability matters and when a probabilistic answer is acceptable.
Notable Quotes
your firewall will work nine out of out of 10 times. Or your authentication will work and will verify who you really are 99% of the times. That might fly once or twice, but then a year later, someone new is going to come in, and he’s going to look at that design and he’s going to go, who the hell accepted this? Niels Tanis · ▶ Watch (2:40)
we are starting to let AI impact the reliability of otherwise quite reliable controls Niels Tanis · ▶ Watch (3:14)
don’t be too harsh on yourself. I’m trying not to be too harsh on myself. I just really have to know and accept whatever the consequences are if I choose to break one of those rules. Niels Tanis · ▶ Watch (11:17)
Key Takeaways
- AI tools degrade security control reliability through non-deterministic outputs.
- Speed from LLM-generated code skips deep problem understanding.
- Combine deterministic and non-deterministic workflows; validate claims technically.
About the Speaker(s)
Niels Tanis has a background in .NET development, pentesting and security consultancy. He is Microsoft MVP and has been involved in breaking, defending and building secure applications. He worked at Veracode as a security researcher on a variant of languages and technologies related to static code analysis and right now is software security engineer at Tidalis. He is married, father of two and lives in a small village just outside Amersfoort, The Netherlands.