SEC536: Adversarial AI - Penetration Testing AI Systems
SEC536Offensive Operations, Artificial Intelligence
SANS Community Benefits
Connect, learn, and share with other cybersecurity professionals
The tools are free. The code is open. The excuse is gone.
4,413 participants→
291 submissions→
123 finalists→
1,775 evaluations→
5 winners
Half the story
The speed. Validated.
Attacks that complete in seconds cannot be answered by investigations that take days. That gap is the entire reason autonomous DFIR exists, and it is also the danger: a fast answer you cannot trust is worse than a slow one. So Find Evil! measured both, and every speed number below survived practitioner verification.
14 min
Full compromised-host investigation, demonstrated live
on the RSA keynote stage with Protocol SIFT. Work that traditionally consumes an analyst’s day.
773 calls
One autonomous Mulder run: 773 logged tool executions
across 11 systems and 120 GB of evidence, every call grounded and auditable. An analyst-weeks workload in a
single committed investigation.
< 1 min
The time it took a judge to independently reproduce a
finalist’s zero-contradiction claim from a clean clone. Verification at the same speed as the
investigation.
The other half
We tried to break our own leaderboard before we trusted it.
Every AI vendor has a benchmark slide. Find Evil! ran the evaluation the industry keeps skipping: practitioners with real evidence, safeguards under live attack, and a dedicated re-verification round before any winner was named. The bar started high: fewer than half of all submissions even survived the turn-in requirements.
ROUND 1
Run it, don't read it
90 practicing incident responders produced 1,775 evaluations and more than 10,000 criterion-level scores against real case data. Documentation earned nothing; execution earned everything.
ROUND 2
Attack the safeguards
Judges attempted to make agents alter the systems under investigation: tamper attempts, path traversal, command injection. Protections had to hold under attack, not in a diagram.
ROUND 3
Re-verify every finalist
Before winners were named, judges reran finalists against evidence the teams had never seen and traced findings back to raw artifacts. Claims that could not be regenerated did not count.
“I ran this myself against a test disk image, and it both worked, and identified 119 of
143 artifacts of compromise.”Luke Pearson, Find Evil! judge, on the first-place winner
“It is actually real and worth trying to get to run with other evidence.”
Jon Stewart, Find Evil! judge
“Genuinely tested, not asserted.”
Shreyas Sriram, Find Evil! judge
“Self-correction is real and log-traceable across multiple runs, and the accuracy report
is honestly self-critical.”Ang See Siang, Find Evil! judge
Speed is why these harnesses exist. Validation is why you can use them. Find Evil! refused to separate the two.
The community reacted
What the field is saying
Verbatim from the public announcement thread.
“The strongest entries did more than produce findings. They showed where
those findings came from, preserved the evidence behind them, and gave the system a way to challenge or disprove
its own conclusions before asking a human to rely on them.”
Sotonye Abam · Find Evil! judge
“They didn’t just find things; they proved them… Protocol SIFT++’s Skeptic
rerunning its tools, TRUDI refuting its own briefing, Mulder’s adversarial phase.”
John W. · Find Evil! judge
“What stood out even more was the emphasis on evidence, reproducibility,
self-correction, and architectural guardrails rather than polished demos alone.”
Yevhen Pervushyn · Find Evil! judge
“Though my entry didn’t place, it was incredibly fun to put AI to
the test against real forensic cases complete with twists and turns. I learned a ton and I can’t wait to
see who wins and how the defensive community as a whole benefits from the advances.”
Jeff Bryner · Find Evil! participant
Join the conversation on the announcement thread.
Coming to the SIFT Workstation
Mulder and TRUDI will ship on SIFT
The first and second place winners are being installed on the SANS SIFT Workstation, the free, open source incident response platform downloaded roughly 60,000 times a year. Pull the next SIFT and the two top-ranked autonomous agents are already on the box.
This is not an integration bolted on after the fact. Every one of the 291 entries was required to run on the SIFT Workstation. The winners were born on the box you are about to download.
- Mulder, first place: five-phase autonomous investigation across memory, disk, network, mobile, and logs.
- TRUDI, second place: hypothesis-driven investigation that refutes its own briefing when the evidence disagrees.
sansforensics@siftworkstation:~$ findevil --list # Find Evil! winners on this workstation ✓ mulder 1st place · 5-phase autonomous DFIR ✓ trudi 2nd place · adversarial self-validation sansforensics@siftworkstation:~$ mulder investigate /cases/host01 read-only mount verified · audit chain armed finding evil...
Every finding these agents record must resolve to a real artifact. Evidence mounts read-only. The audit chain logs everything.
Where it started
One question from the RSA keynote stage
It began when Rob T. Lee, Chief AI Officer and Chief of Research at SANS Institute, took the RSA Conference keynote stage and demonstrated Protocol SIFT: AI agents driving the SIFT Workstation through a full compromised-host investigation in 14 minutes, work that traditionally consumes an analyst’s day.
The audience reaction was consistent, and it was not about speed. It was trust. Nobody doubted the agent could run the tools. They doubted whether a responder could act on what it reported in the middle of a live incident.
Lee identified that gap as the field's central unanswered question and built Find Evil! to answer it, convening the community, the judging panel, and Protocol SIFT itself as the open source baseline every single entrant was required to extend. The challenge ran April 15 to June 15, 2026, with more than $22,000 in prizes, free entry, and one rule above the rest: agents had to prove their findings, not just produce them.
“The question was never ‘is it fast.’ It was ‘would I act on what it just
told me at two in the morning.’ You answer that by handing the tools to 90 responders and letting them try
to break them.”
Rob T. Lee · Chief AI Officer and Chief of Research, SANS Institute
What these are
Not models. Harnesses.
Each winner wraps a frontier language model in a scaffold that makes its reasoning safe to trust during an incident. The model drives. The rails hold.
The challenge rules demanded it: every entry had to correct its own errors without human intervention, trace every finding to a specific artifact, file, offset, or log entry, and reason like an investigator, producing a structured narrative rather than a raw execution log. Harnesses are what that looks like when it is enforced.
Frontier LLM forms hypotheses, decides what to examine next, writes the narrative
The harness rails read-only evidence · typed, allowlisted tools · no shell · findings must resolve to real artifacts · SHA-256 audit chain
Toolset memory, disk, timeline, registry, network and log tooling, dozens to hundreds per agent
The results
Five winners. All open source. All yours.
Verified by practitioners, re-verified before the announcement, and released under open source licenses for anyone to run, audit, and improve.
First place
Mulder Ships with SIFT
The most thoroughly judged agent in the field, and the one a judge ran on his own evidence.
145 typed tools120 GB / one run11 systems773 logged calls
“This is the most impressive entry I’ve seen. It is actually real.” Jon Stewart, Find Evil! judge
Second place
TRUDI Ships with SIFT
Threat Response Unit for Digital Investigation. The agent that caught itself being wrong.
250+ tools8-host APT case15+ server gatescausal-chain logs
“Honestly one of the best submissions I reviewed across the board.” Brett Cumming, Find Evil! judge
Third place
Camel
Code-mode forensics with the broadest verified tool surface in the field.
code-mode MCPfinding → exact commandmulti-host casesandboxed runtime
“This was the strongest submission I reviewed.” Joshua McCray, Find Evil! judge
Fourth place
FindEvil
Deep Linux incident response, with read-only enforced by the build itself.
45 typed tools98.6% recall / 552 attacks102 security testsAST-gated writes
“The best submission in the pool by a meaningful margin.” Hardik Chawla, Find Evil! judge
MIT license
github.com/marlyocat/findevil
Fifth place
Protocol SIFT++
Extends the official baseline, then reruns its own tools to try to prove itself wrong.
Investigator + Skeptic14/14 tampers refusedreplayable runsguardrails provable, no key
“The most credible self-correction in this pool.” Rathan Ramachandra, Find Evil! judge
In their own words
What 90 practitioners kept saying
1,775 evaluations produced hundreds of pages of judge commentary. These lines kept appearing.
“The hallucination precautions and sources being required and challenged are
fantastic. I can see this one going somewhere.”
Heather Barnhart, Find Evil! judge and SANS Fellow
“The most impressive project I evaluated so far. Impressive demonstration at
a 120 GB dataset scale.”
Yotam Perkal, Find Evil! judge
“A brilliant, battle-hardened design pattern: LLM at the edge, determinism
inside.”
Devashri Datta, Find Evil! judge
“An HMAC key held off the agent means it physically cannot forge a
verdict.”
Kenneth G Hartman, Find Evil! judge
“A finding only becomes valid if it was actually read from evidence, not
guessed.”
Saitrik Chakraborty, Find Evil! judge
“The public retraction of a named false positive is the gold standard of
honesty this competition rewards.”
Saitrik Chakraborty, Find Evil! judge
“It genuinely reinvestigated and withdrew four of its own false
positives.”
Sotonye Abam, Find Evil! judge
“Best in class of those reviewed to date. Honest, correct, and well
documented.”
Cheri Carr, Find Evil! judge
“Unsupported CONFIRMED claims are refused by code, not merely discouraged by
prompt.”
Yevhen Pervushyn, Find Evil! judge
Why this matters
Four things Find Evil! just changed
01
The machines learned to say “I was wrong.”
The trait practitioners rewarded most was self-refutation. TRUDI refuted its own briefing when the evidence came back empty. Protocol SIFT++’s Skeptic retracted a rootkit finding it had previously confirmed. Another finalist reinvestigated and withdrew four of its own false positives. In a field terrified of AI hallucination, the winners were the agents that argued with themselves.
An incident responder’s most valuable trait, engineered into software.
02
Honesty outscored perfection.
Teams that documented their own misses beat teams claiming flawless accuracy, consistently. One winner found and fixed two security bugs in its own guardrails and published the commits. Judges called that candor the standard the competition rewards, and the scores backed them up.
The incentive the AI industry keeps getting backwards, inverted in public.
03
Guardrails are table stakes now.
Nearly every serious entry enforced read-only evidence access architecturally: typed tools, no shell, findings that must resolve to real artifacts. The question moved from “can we stop it fabricating evidence” to “how deep can it investigate.” That is a solved-problem milestone for the field.
Depth of analysis, not tool access, is the scarce skill now.
04
Trust became a benchmark.
“Would a responder act on it at two in the morning” stopped being a rhetorical question and became something 90 practitioners measured, attacked, and re-verified at field scale. The methodology is public. The yardstick belongs to the community now.
The two-in-the-morning question finally has a methodology. It will not be the last time it is asked.
Take this with you
Buying an autonomous agent? Ask these five questions.
Find Evil! did not just rank hackathon entries. It established, in public, what good looks like. Every question below broke real submissions, and every one has an open source answer you can hold your vendor against.
01
Can it prove a finding, or just cite one confidently?
The winners enforce this structurally: a finding that does not resolve to a real file, line, or hash cannot be recorded at all.
02
What happens when you attack it?
Tamper attempts, path traversal, command injection. Ask for the test results, not the architecture diagram. Find Evil! judges attacked every finalist; protections had to hold, not be described.
03
Has it ever retracted its own finding? Show the log.
The trait practitioners rewarded most was self-refutation. The second-place winner refuted its own briefing. The fifth-place winner retracted a rootkit finding it had previously confirmed. If the vendor’s agent has never said “I was wrong,” it has never been pushed.
04
Where is the honest accuracy report?
Misses, false positives, and limits, in writing. Across 123 finalists, teams that documented their own failures outscored teams claiming perfection. Every winner ships one.
05
Can an independent party rerun it and get the same findings?
Before winners were named, judges reran finalists on evidence the teams had never seen. Reproducibility on unseen data is the whole game. Anything less is a demo.
The five winners answer all five questions in public repositories, free. Anyone selling you an autonomous agent should answer them too.
First they said AI would replace analysts.
Then 90 analysts graded the AI.
Then 90 analysts graded the AI.
Run them yourself
The tools are free. The code is open. The excuse is gone.
Download the SIFT Workstation and the top two winners come with it. Clone the rest, point them at your own evidence, and improve them. That is the point of doing this in the open.
The SIFT Workstation and Protocol SIFT are free and open source. Winner results were certified following an independent validation round.