SEC536: Adversarial AI - Penetration Testing AI Systems




In-Person
In-Person & Virtual
In-Person & Virtual
Speaker: Jason Higley
In-Person & Virtual
In-Person & Virtual
In-Person & Virtual
Vishing attacks targeting Google accounts have grown more sophisticated — not by breaking authentication systems, but by weaponizing them. This talk presents a forensic case study of a real, coordinated attack that combined display-name header spoofing with deliberate abuse of Google's account recovery notification infrastructure to deliver a fully authenticated, DKIM-signed Google security alert directly to the victim's inbox.
The attack began with a phone call impersonating Google Support, followed by two precisely timed emails. The first was a spoofed phishing lure — appearing as "support@google" while originating from attacker-controlled infrastructure at case-priority.com, relayed through MailChannels and Hostinger, with SPF passing on the envelope sender while the visible From header pointed to a non-existent domain. The second required no spoofing at all: the attacker added the victim's address as a recovery contact on a throwaway Gmail account, then triggered account changes — causing Google's own servers to send a legitimate security alert to the victim's inbox. The contrast between the two emails is forensically significant and operationally underappreciated.
Attendees will learn to identify SPF alignment failure behind a passing SPF result, read SMTP relay chains to surface attacker infrastructure, and distinguish display-name spoofing from envelope spoofing in raw headers. The talk concludes with a MITRE ATT&CK mapping of the full attack chain, header-level IOCs, and hardening recommendations for Google Workspace environments that DFIR practitioners can implement immediately.
In-Person & Virtual
Speaker: Bryson Byrd
Every ransomware investigation begins long before encryption. Success depends on recognizing the seemingly ordinary activity attackers use to establish access, move through an environment, and position themselves before launching the final stage of an attack. Using a real-world incident involving Fusion Computing, presenters reconstruct an identity-driven intrusion that was identified and contained before ransomware could be deployed.
Attendees will follow the investigation from an initial social engineering phone call through PowerShell execution, command-and-control activity, and living-off-the-land techniques that initially blended with legitimate administrative behavior.
Bryson demonstrate how endpoint telemetry, identity signals, and forensic artifacts were used to reconstruct the attack timeline, validate attacker activity, identify the evidence that mattered most, and prioritize containment decisions before encryption could occur. The session also examines how the observed techniques map to the MITRE ATT&CK framework, highlighting practical opportunities to improve visibility into identity-based attacks and strengthen threat hunting workflows.
Participants will leave with practical hunting techniques for detecting identity-driven intrusions, investigative approaches for validating attacker activity, and recommendations for strengthening ransomware detection before encryption begins.
In-Person & Virtual
Microsoft OneDrive Offline Mode stores significantly more than synchronized files. Modern versions of the client cache data from Outlook, Microsoft Teams, and SharePoint, creating a rich source of forensic evidence that is often absent from traditional acquisition and analysis workflows. Investigators who understand these artifacts can recover valuable context about user activity even when cloud access is unavailable.
This session provides a deep technical examination of OneDrive Offline Mode artifacts and demonstrates how to identify and analyze files associated with email, calendar events, meeting invitations, Teams chats, and SharePoint data. The presentation explains where these artifacts are stored, how they relate to one another, and techniques for reconstructing user activity across Microsoft 365 services.
The session also examines OneDrive's Optical Character Recognition (OCR) metadata, showing how text extracted from images and documents can become valuable investigative evidence while also introducing privacy and security considerations. Attendees will learn how OCR data is stored, what information can be recovered, and the implications for incident response, insider threat investigations, and enterprise data protection.
Using real artifacts and practical demonstrations, attendees will gain a repeatable methodology for locating, parsing, and correlating OneDrive Offline Mode data. They will leave with actionable techniques that can be immediately incorporated into forensic collections, artifact triage, and Microsoft 365 investigations.
In-Person & Virtual
Speaker: Abdul Khadir
Threat hunting has become a critical capability for organizations facing increasingly sophisticated ransomware attacks. While many security teams rely heavily on automated detection tools, advanced adversaries continue to evade traditional controls by exploiting identity systems, abusing legitimate administrative tools, and moving laterally across hybrid IT and operational technology (OT) environments.
This session presents a practical, intelligence-driven threat hunting methodology that enables defenders to proactively identify attacker behaviors before ransomware deployment.
Drawing from real-world experience securing global manufacturing environments, the presentation demonstrates how organizations can build effective threat hunting programs by combining endpoint telemetry, identity analytics, network traffic analysis, SIEM, XDR, threat intelligence, and behavioral analytics. The session will cover hypothesis-driven hunting, mapping attacker techniques to the MITRE ATT&CK framework, detecting living-off-the-land (LOTL) activity, identifying early indicators of ransomware campaigns, and prioritizing hunts using risk-based intelligence.
Attendees will learn practical hunting workflows, detection engineering techniques, and investigation strategies that can be implemented using commonly deployed enterprise security platforms. The presentation will also discuss lessons learned from enterprise environments, common mistakes organizations make during ransomware investigations, and recommendations for improving detection coverage and incident readiness.
Key Takeaways:
In-Person & Virtual
The architectural advancements to Siri and Apple Intelligence as part of iOS 27 and macOS 27 fundamentally alter the nature of user-device interaction. The OS is no longer a passive tool waiting for commands, but it has transitioned to a system capable of autonomous, context-driven action on behalf of the user. The next-generation Apple Intelligence and Siri AI possess onscreen awareness, contextual understanding, and read/write access to the user's data.
When an operating system natively summarizes user documents, alters the tone of text including messages, or generates almost real looking synthetic images, the difference between human-authored and algorithmic generation becomes blurry to the investigators.
This talk establishes a forensic framework for analyzing Apple Intelligence and reveals the undocumented forensic artifacts left behind by the next-generation Apple Intelligence and the entirely rebuilt Siri AI on iOS27 and macOS 27. We will perform a deep dive into the specific, undocumented file system hierarchies where prompt histories live, how "deleted" AI sessions persist, and the Apple Intelligence logs that commercial tools currently miss.
Attendees will leave with an open-source tool and an evidence-based methodology to identify human-authored, AI-generated, and AI-modified content.
In-Person & Virtual
Speaker: Michał Sołtysik
Modern Endpoint Detection and Response (EDR) platforms heavily publicize their ability to block ransomware through real-time behavioral correlation, file-system telemetry, and API hooking. However, these detection layers rely on predictable heuristic metrics, such as high-frequency cryptographic API calls and linear directory traversing.
This presentation demonstrates how progressive structural changes can completely blind enterprise defenses without changing the operational outcome of a simulation. Through an iterative, 12-stage analysis of a purpose-built evaluation agent, we map exactly how signature scanners and behavioral engines fail when confronted with execution-delay jitter, directory-walk elimination, and user-space stream transforms.
We will reveal how shifting from standard API frameworks (like .NET CryptoStream) to custom, user-mode byte manipulation (ProcessFileStealth) easily neutralizes static heuristics. Furthermore, we demonstrate how avoiding recursive directory loops via flat enumeration techniques undercuts an EDR's correlation logic. Attendees will walk away with concrete data on engine blind spots and practical testing methodology to validate their own defensive posture against low-and-slow execution vectors.
In-Person & Virtual
In-Person & Virtual
Speaker: Jason Trapp
"What data was taken?" is one of the first questions legal counsel asks during a ransomware engagement, and in most cases, it's the hardest question to answer. Rclone has become the de facto exfiltration tool used by ransomware affiliates: it's free, flexible, and quiet. Analysts typically fall back on firewall logs for a byte count however, this doesn’t tell us what has been taken.
This session demonstrates how to turn a memory image, which can be paired with a packet capture, into an inventory of exfiltrated files. It dissects what rclone leaves in memory, including execution artifacts, command line arguments, and the config file. These details include the deobfuscated remote location, which reveals the destination service, the account, and the master key. From the packet capture, the tool carves the encrypted uploaded blobs, collects the exact file sizes, and then correlates them with the memory artifacts, source directory listings, and file sizes. This session shows why capture timing affects what you can collect: account keys, file encryption keys, and transport keys. During this session, a free tool that automates this and formats the output for executives and legal counsel to help drive notification decisions as well as the limitations of this tool.
In-Person & Virtual
In-Person
Speakers: Samuel Addington & Christopher Earl
In July 2025, Check Point Research disclosed MCPoison (CVE-2025-54136, CVSS 7.2), a flaw in Cursor IDE (versions ≤ 1.2.4) where the Model Context Protocol trust model bound approval to an MCP server's name rather than its command or arguments. An attacker commits a benign .cursor/rules/mcp.json to a shared repository; a developer approves it once; the attacker later swaps in a reverse shell that runs silently on every project open — persistent RCE with no second prompt.
A companion flaw, CurXecute (CVE-2025-54135, CVSS 8.6), reached the same outcome through prompt injection that wrote to ~/.cursor/mcp.json and executed even after the user rejected the change.
These two incidents expose the problem this session tackles: when the acting entity is an AI agent, the authentication event is legitimate, the spawning process is the expected runtime, and the action falls within the agent's granted permissions so conventional EDR timelines show nothing anomalous, and the triggering instruction frequently lives only in ephemeral prompt context that does not survive default log retention.
The session reproduces both CVEs live on an isolated host, then contrasts their forensic footprints: MCPoison leaves recoverable on-disk evidence (git history of the config, file modification times, file-integrity-monitoring hits), while the prompt-injection path leaves almost nothing once the session closes. From that contrast the presenter derives a NIST-mapped incident-response playbook for agentic systems what to preserve before containment, how to triage runtime-escape versus privilege-boundary-escape versus reasoning-compromise incidents, and how to instrument MCP tool-call telemetry so the originating instruction is captured as evidence rather than lost.
Attendees leave with the lab build (vulnerable and patched configs), a forensic-artifact collection checklist for MCP and agent incidents, file-integrity-monitoring and detection logic for mcp.json tampering, and an open-source telemetry wrapper that records agent tool calls and their originating context in a retention-safe format.
In-Person & Virtual
Speaker: Javid Bakhtiani & Sarah Bukhari
When a manufacturing organization with two physical sites was compromised through an unpatched FortiGate 60E firewall running a vulnerable FortiOS 7.2.2 build affected by CVE-2022-42475, alongside a set of end-of-life Windows Server 2012 hosts, the incident response team uncovered far more than encrypted files. The environment had a fully flat network with switch loopbacks, no DHCP failover, and backups that had never been audited, meaning entire servers were missing from the backup set without anyone knowing.
This session walks through the full incident lifecycle: initial access through the unpatched firewall and legacy servers, confirmed data exfiltration alongside encryption, forensic preservation through full host imaging and triage collection before any remediation began, and the decision to pursue a decryptor rather than risk an estimated 50 percent data loss across the environment, including Active Directory.
Attendees will see the exact hardening checklist used across both sites, covering network containment, Active Directory and Kerberos resets, GPO and credential audits, and EDR plus anti-ransomware deployment, along with the specific gaps in the client's prior security posture that allowed the incident to happen in the first place.
This talk is built for both technical responders who want the operational detail and less technical attendees who need to understand how a single unpatched device cascades into a two-site recovery effort.
In-Person & Virtual
Speakers: Justin Price & Rakesh Kumar Narsingoju
Enterprise data exfiltration rarely presents as a clear, high-confidence alert. Instead, investigators must determine whether observed activity reflects legitimate business use or policy-violating or malicious behavior using incomplete and often ambiguous telemetry. This session presents real-world DFIR investigations involving insider-driven data movement across removable media, email, and cloud channels, where initial detections alone were insufficient to determine risk or intent.
The talk examines multiple investigation patterns observed in enterprise environments, including long-duration accumulation of data to removable media, burst exfiltration events where most transfers occur within a short timeframe (such as hours preceding user termination), and multi-channel movement combining USB transfers, email forwarding to personal accounts, and cloud-sharing activity.
Additional scenarios include misuse of legitimate access (where data aligns with a user’s role but is accessed or transmitted outside expected context) and repeated low-volume transfers designed to evade threshold-based detections. Attendees will follow the investigation sequence used to expand alerts into full-scope cases, including correlating endpoint telemetry such as DeviceFileEvents for removable media activity, email telemetry such as EmailEvents and attachment metadata, and identity signals related to account activity and user lifecycle changes. The session demonstrates how investigators perform historical lookbacks, validate data sensitivity through targeted sampling, and establish behavioral baselines to identify anomalies in volume, timing, and access patterns.
The presentation highlights a critical limitation in traditional DFIR workflows: activity-based detections alone cannot distinguish between legitimate use and data exfiltration without contextual and behavioral analysis. To address this, the session introduces a repeatable investigation methodology that includes classification of exfiltration patterns (slow accumulation, burst transfer, and multi-channel movement), a correlation framework linking endpoint, email, and cloud telemetry, and a decision model for assessing user intent in ambiguous scenarios.
Attendees will leave with practical, directly applicable outputs, including an investigation checklist for multi-channel data movement, behavioral indicators for identifying suspicious activity, and detection and correlation approaches that can be adapted within platforms such as Microsoft Defender XDR or similar environments.
In-Person & Virtual
In-Person & Virtual
In-Person & Virtual
In-Person & Virtual
In-Person
In-Person
In-Person
In-Person
In-Person
After a full day of DFIR insights, it is time to step away from the screens and join us upstairs at The View for the SANS Social. Take in breathtaking panoramic views of Washington, DC while enjoying drinks, food, and entertainment with fellow attendees, speakers, and SANS experts. It is the perfect chance to unwind, continue the day’s conversations, make new connections, and enjoy a great evening with the DFIR community.
In-Person
In-Person
In-Person & Virtual
In-Person & Virtual
Traditional endpoint detection and response (EDR) agents are powerful, but what happens when threat actors bypass them entirely by spinning up their own unmanaged systems within your hypervisor? Unit 42 investigated a complex intrusion where the notorious threat group Scattered Spider (aka Muddled Libra, UNC3944) did exactly that. nd older Outlook versions on Reddit.
After leveraging social engineering to gain administrative credentials, the threat actors accessed the victim's VMware vSphere portal, created an unmanaged rogue virtual machine, and utilized it as an operational beachhead to conduct actions on objectives.
Fortunately, the security team terminated the attackers' access before they could delete the rogue VM, leaving behind a highly valuable forensic record. This presentation delivers a forensic reconstruction of Scattered Spider’s active playbook built directly from the artifacts recovered from that beachhead system. We will walk through the exact 15-hour timeline, detailing the group's living-off-the-land tradecraft and distinct troubleshooting behaviors.
Attendees will learn the technical mechanics of:
• Establishing persistence using an SSH tunnel hosted in an attacker-controlled AWS S3 bucket.
• Disabling and powering down virtualized domain controllers via vSphere, mounting their VMDK disks, and copying the NTDS.dit database and SYSTEM registry hive to decrypt and harvest domain-wide credential hashes.
• Executing Active Directory enumeration tools, and specifically targeting various computing services to map critical infrastructure.
• Navigating SaaS cloud databases and attempting to exfiltrate sensitive Outlook PST files via the S3 Browser tool.
We will also analyze the threat actor's real-time struggles documented in their browser history: searching Bing for how to bypass blocked file-sharing sites, troubleshooting Outlook speed issues, and researching how to find older Outlook versions on Reddit.
In-Person & Virtual
In-Person & Virtual
Ransomware is bad, but at least you get choices. Pay, do not pay, restore from backup, call the FBI. Bad options, but options. Cloud and AI hijacking does not work like that. There is no decryptor, nobody to negotiate with, no wallet to trace. The attacker gets what they came for, the provider gets paid, and you get the invoice.
Hijacking other people's machines is not new, but what it is worth has changed. In the 2000s it was sabotage and botnets. By the 2020s it was crypto mining, stealing compute to mine coins. Now it is AI: a stolen cloud key buys frontier model access, and that is not just a bill, it is free R&D. The most interesting buyers are the ones who cannot get it legitimately, actors under sanctions or export controls who now have a way to rent capability they are embargoed from buying.
I respond to these for a living. I will walk two anonymized cases and what they cost: a GCP cryptojacking job off a leaked service account that ran into the tens of thousands in hours, and an LLMjacking case on a stolen key that reached six figures before anyone noticed. For a small company either one is not a dispute, it is the end of the business. Then the harder question: as defenders we are failing at this, and I will be specific about why, what the response actually looks like when there is no host and no malware to grab, and the GCP, AWS and Azure cheat sheet you can take back to the SOC.
In-Person & Virtual
This talk will delve into a highly sophisticated Chinese APT campaign, tracked as CL-STA-0049, that has been targeting governments, defense, telecommunication, education and aviation sectors in Southeast Asia and South America since at least March 2023. This campaign demonstrates a high level of complexity, utilizing a multi-stage attack chain and employing advanced evasion techniques to bypass traditional security measures.
At the heart of this operation lies "Squidoor", an advanced backdoor exemplifying evolving state-sponsored tactics. This sophisticated backdoor features both Windows and Linux versions, employing an unprecedented array of 10 different C2 communication methods, including rarely seen techniques like abusing Outlook for C2 communication.
In the presentation, we will first share the different TTPs the threat actor used to gain access to the victim network and deliver the "Squidoor" malware.
Second, we will provide a thorough and detailed technical malware analysis of the "Squidoor" malware. We will begin with the execution flow of the malware involving LOLBAS and shellcode. Next, we will dive into the 10 different C2 communication methods Squidoor is able to utilize and explain how they work. And finally we will touch upon Squidoor's different capabilities and encryption protocols it's utilized as part of the c2 communication.
We will end the presentation with giving the audience practical ways to hunt for the Squidoor malware. By the end of the talk, attendees will gain a unique glimpse into the inner working of sophisticated Chinese APT tooling and operations.
In-Person & Virtual
AI-assisted IDEs and coding tools are becoming part of normal developer workstation activity, but most DFIR playbooks still treat them as background applications rather than forensic evidence sources. This creates a visibility gap during developer compromise and software supply chain investigations: responders may see file changes, package installs, or repository activity, but miss the local session context, tool activity, workspace state, and configuration that explain how those actions occurred.
This session presents a DFIR-focused artifact study of AI-assisted developer environments such as VS Code with GitHub Copilot, Cursor, Claude Code, Codex, and related IDE- or terminal-based coding assistants. The research will compare the local traces these tools can leave behind, including session history, workspace context, prompts, tool outputs, command traces, debug logs, extension data, configuration files, local observability records, and repository-level instructions. It will also examine how these environments intersect with traditional attacker targets such as environment variables, API tokens, SSH keys, credential stores, package-manager credentials, and source-control authentication.
The talk will map these artifacts to realistic investigation scenarios involving malicious packages, suspicious editor or extension behavior, credential exposure, risky activities, and downstream CI/CD abuse. The focus will be on how these artifacts can be parsed, what important activity can already be observed through current EDR, endpoint, editor, source-control, and enterprise telemetry, and where visibility gaps remain for responders investigating compromised developer workstations.
The session will also discuss open-source tooling opportunities to make this analysis easier: lightweight collectors, artifact parsers, timeline builders, and triage helpers focused on AI-assisted developer environments. Attendees will leave with a practical artifact map, telemetry checklist, monitoring considerations, and investigation ideas for developer workstations that may serve as an initial access path into the software supply chain.
In-Person & Virtual
Identity has become one of the most targeted attack surfaces in modern enterprises. Instead of stealing passwords or deploying malware, attackers increasingly obtain valid OAuth access tokens and reuse them to move through SaaS applications and cloud environments. Unfortunately, many organizations lack the telemetry and detection logic needed to identify token abuse before it results in account compromise.
This session demonstrates how defenders can hunt for OAuth token theft and replay using identity provider logs, API telemetry, cloud audit logs, and endpoint evidence. Through realistic attack scenarios—including token replay, bearer token theft, and OAuth redirect abuse—attendees will learn how to build hunting hypotheses, identify meaningful indicators, and create practical detections.
The presentation includes example KQL and Sigma detections, recommended logging sources, ATT&CK mappings, and a reusable hunting playbook that attendees can adapt to Microsoft Entra ID, Okta, Google Workspace, and other identity platforms.
In-Person & Virtual
Speaker: Alexander Holcomb
The Apple Unified Log has become a critical forensic artifact that DFIR analysts leverage when investigating compromised Apple devices. However, the binary structure format of the Unified Log is undocumented by Apple and remains a black box for many analysts and users. This forces analysts to rely on native Apple tooling. To better understand how a log message is assembled, we must understand how the log message is reconstructed from fragmented data across a system.
This talk will be a deep dive into the binary structure of the Apple Unified Log format and a walkthrough of the three binary components of the Unified Log. We will discuss each binary file format and how each component leverages each other in order to assemble a full log message. In addition, the talk will discuss some challenges that may be encountered when parsing the log formats.
Finally this talk will discuss a cross-platform, open-source Rust library designed to parse the Apple Unified Log files.
In-Person & Virtual
Speaker: Luis Garcia
Hypothesis-driven hunting is often presented as a process of generating hypotheses and translating them into hunt queries. In practice, the most difficult part begins after the query runs. A single hunt can return thousands or tens of thousands of events, leaving analysts to manually baseline activity, identify outliers, and determine which findings deserve investigation.
This presentation shares lessons learned from building and deploying a production LLM-agent workflow focused on hunt execution and analysis. Rather than stopping at query generation, the system assists analysts throughout the investigation process by iteratively refining hunt results, building behavioral baselines, clustering similar activity, identifying anomalies, and surfacing the most promising leads for human review.
The talk walks through a complete threat hunting case study, showing how an initial broad hunt query evolves through multiple stages of automated analysis. Attendees will see how specialized agents perform result triage, compare activity against environmental baselines, identify unusual patterns, generate investigative pivots, and summarize findings with supporting evidence. The workflow is demonstrated against real security telemetry and production hunting scenarios.
The presentation also examines the challenges encountered in practice, including weak baselines, hallucinated conclusions, inconsistent results across model versions, and analyst trust. We discuss the safeguards implemented to ensure findings remain explainable, evidence-based, reproducible, and subject to analyst validation.
Attendees will leave with a practical framework for using LLM agents to augment hunt execution, reduce manual triage effort, and help analysts focus on the most suspicious activity without sacrificing transparency or investigative rigor.
In-Person & Virtual
Speakers: Abdalla Jamal Alshamsi & Captain Ahmad Tariq Almrashda
Cyber-enabled financial crimes rarely begin with sophisticated malware or advanced intrusion techniques. More often, they begin with trust. This presentation examines a real-world investigation that started with a carefully crafted social engineering campaign and phishing attack that successfully deceived the victim, resulting in unauthorized access to accounts, compromise of digital assets, and the theft of cryptocurrency.
What initially appeared to be a routine phishing incident rapidly evolved into a complex, multi-stage investigation involving digital forensics, attack reconstruction, blockchain intelligence, asset tracing, legal coordination, and international cooperation.
The session follows the complete investigative timeline from the initial victim report through the final operational outcome. Attendees will learn how investigators reconstructed the attacker’s actions, identified key forensic artifacts, analyzed compromise indicators, and established a timeline of events from the initial social engineering contact to the unauthorized transfer of digital assets.
The presentation will demonstrate how technical evidence collected from compromised systems, transaction records, and digital artifacts was correlated to identify investigative leads and support attribution efforts. Particular focus will be placed on the challenges investigators faced while tracing stolen cryptocurrency through multiple transfers intended to obscure ownership, frustrate analysis, and delay recovery efforts.
Attendees will also gain insight into the operational realities of cryptocurrency investigations, including evidence preservation, blockchain analysis methodologies, attribution challenges, cross-border information sharing, and the integration of forensic findings into judicial processes.
The investigation ultimately resulted in the identification of the suspect, the freezing of criminally controlled digital assets, the recovery of assets associated with the offense, the issuance of judicial orders, and the successful apprehension of a suspect located outside the victim’s jurisdiction through established legal procedures. Rather than presenting a success story alone, this session provides a practical and repeatable investigative framework for DFIR practitioners, cybercrime investigators, analysts, and law enforcement professionals.
Attendees will leave with actionable methodologies for investigating cryptocurrency-enabled crimes, tracing digital assets, supporting recovery operations, and transforming technical findings into operational outcomes.
In-Person & Virtual
Speaker: Molly Correia Modern SOC and threat hunting workflows are increasingly augmented by large language models for alert triage and hypothesis generation, yet these systems remain unreliable in practice. They frequently produce confident but incorrect classifications, including overestimating routine activity as active compromise, mislabeling benign scheduled operations as command-and-control behavior, and incorrectly mapping post-exploitation or vulnerability exploitation stages.
For example, legitimate enterprise behavior such as backup jobs running at 2 AM may be incorrectly interpreted as beaconing or data exfiltration patterns. These hallucinations increase analyst workload, inflate false positives, and introduce operational risk by obscuring real threats.
This presentation introduces a self-correcting AI threat hunting pipeline designed to address these limitations through structured feedback, institutional context modeling, and persistent learning. A key extension of this work is the concept of *institutional benchmark engineering*, where organizations define their own environment-specific ground truth. This allows security teams to encode what “normal behavior” looks like in their own infrastructure, for example, scheduled backups, patch deployments, or batch ETL jobs so that these are not misclassified as malicious activity.
The system is evaluated on a custom benchmark of 30 realistic security scenarios derived from publicly documented attack patterns and detection use cases.(CISA KEV and NVD CVE) Each scenario includes structured ground truth for severity, recommended response action, and MITRE-aligned attack categorization.
A baseline evaluation using Gemini-2.0-Flash achieved 33.3% accuracy, revealing significant weaknesses across severity classification, response action selection, and attack categorization. The system then applies a persistent learning loop that captures failures, stores them as structured JSON memory, and generates targeted course corrections that evolve the prompt across iterations.
Following a single learning cycle, performance improved to 96.7% accuracy, reducing failures from 20 scenarios to a single remaining error. The loop injected 43 targeted corrections addressing systematic issues such as DDoS over-classification, infiltration mislabeling, and incorrect remediation actions.
The session walks through the full architecture, including benchmark design, institutional baseline modeling, failure-memory construction, automated correction generation, and iterative prompt evolution. It also demonstrates how organizations can continuously refine detection intelligence by embedding site-specific operational context directly into evaluation benchmarks, thereby reducing hallucinations without model retraining.
Attendees will receive:
• Full open-source Python implementation of the persistent learning loop
• Framework for building institutional SOC benchmarks tailored to enterprise environments
• Benchmark generation pipeline with KEV-derived scenarios
• Failure-memory and correction injection patterns for self-improving SOC workflows
• Prompt engineering strategies for deploying self-correcting AI in security operations
This talk provides practical methods for detection engineers and threat hunters to systematically evaluate, contextualize, and improve the reliability of AI-driven security decisions before operational deployment.
In-Person & Virtual
In-Person & Virtual
Speaker: Antonia Feffer
Traditional MITRE ATT&CK coverage assessments typically treat detections as binary, asking only whether an analytic exists for a given technique. This approach fails to capture whether a detection is resilient to implementation variations, adversary evasion, or alternative attack paths. As part of MITRE CTID's Summiting the Pyramid initiative, the presenters developed an "effective coverage" calculator that evaluates detection quality by measuring how well analytics perform across multiple technique implementations and how robust they remain against adversarial evasion opportunities.
This session will present the research methodology behind the calculator, including the introduction and analysis of ATT&CK technique implementations, robustness and precision evaluation, and the framework used to quantify effective detection coverage. The presenters will demonstrate the calculator tool against a variety of Elastic and Sigma analytics and apply it to a real-world 2020 intrusion case study, showing how the methodology identifies gaps that traditional ATT&CK coverage metrics fail to expose and helps prioritize improvements to existing detections. Attendees will leave with a practical framework for evaluating detection effectiveness beyond binary ATT&CK coverage, access to the effective coverage calculator, and guidance for applying the methodology to improve detection engineering and threat hunting programs within their own environments.
In-Person & Virtual
In-Person
In-Person
In-Person
Registration:
About DFIR NetWars: Focused on digital forensics, incident response, threat hunting, and malware analysis, this tool-agnostic approach covers everything from low-level artifacts to high-level behavioral observations.
Computer Requirements: Laptop/desktop-based
Extra Requirements: Files downloads are required to participate.
Recommended For: Experienced Digital Forensic Analysts, Forensic Examiners, Media Exploitation Examiners, Malware Analysts, Incident Responders, Threat Hunters, Security Operations Center (SOC) Analysts, Law Enforcement Officers, Federal Agents, Detectives, and Cyber Crime Investigators.
Disciplines: Digital Forensics, Incident Response.
Example Topics:
Interactive Scenario: As a DFIR specialist, you are provided with evidence files from a series of mysterious compromised systems and conventional computing environments. Your mission? Use your DFIR skills to shed light on attack vectors, indicators of compromise, and other evidence needed to resolve the incident.
In-Person & Virtual
Registration:
About DFIR NetWars: Focused on digital forensics, incident response, threat hunting, and malware analysis, this tool-agnostic approach covers everything from low-level artifacts to high-level behavioral observations.
Computer Requirements: Laptop/desktop-based
Extra Requirements: Files downloads are required to participate.
Recommended For: Experienced Digital Forensic Analysts, Forensic Examiners, Media Exploitation Examiners, Malware Analysts, Incident Responders, Threat Hunters, Security Operations Center (SOC) Analysts, Law Enforcement Officers, Federal Agents, Detectives, and Cyber Crime Investigators.
Disciplines: Digital Forensics, Incident Response.
Example Topics:
Interactive Scenario: As a DFIR specialist, you are provided with evidence files from a series of mysterious compromised systems and conventional computing environments. Your mission? Use your DFIR skills to shed light on attack vectors, indicators of compromise, and other evidence needed to resolve the incident.
In-Person & Virtual