Remember when ChatGPT was going to revolutionize your security program overnight? Two years into production deployments, it's time for an honest conversation about what actually happened when LLMs met the reality of 24/7 security operations, application security and cyber threat intelligence.
This talk cuts through the vendor hype to share real-world experiences integrating AI into security workflows. We'll explore specific use cases where LLMs genuinely improved security—from automating alert triage to synthesizing threat intelligence—and more importantly, where they failed spectacularly. You'll learn why that "AI-powered code improvement recommendations" POC crashed and burned, and how hallucinating LLMs almost caused priority one incidents.
The session deep-dives into the risks nobody wants to discuss: what happens when your incident data trains someone else's model, how attackers can manipulate AI-driven security decisions through prompt injection, and why relying on AI to triage alerts can either be the best or worst decision you make this year. We'll outline battle-tested guardrails including data handling requirements, access control strategies, and the critical "human-in-the-loop" checkpoints that separate success from disaster.
Attendees will leave with a practical framework for evaluating AI integration in their own SOCs, a clear understanding of the security and privacy implications, and most importantly, realistic expectations about what AI can and cannot do for a security program. Whether you're actively using LLMs, still evaluating, or have been told "absolutely not" by legal, you'll gain actionable insights from our year-long journey of successes, failures, and lessons learned the hard way. No vendor pitches, no hand-waving about AGI solving all our problems—just practical experiences from the trenches of running AI-augmented security program.
Key Takeaways
- Real metrics on time savings and accuracy from production LLM deployments
- Specific use cases that work (and those that definitively don't)
- Critical security guardrails for protecting sensitive data in AI workflows
- A decision framework for evaluating AI tools without falling for hype.
