The Benchmark Nobody Asked For — But Everyone Needed
SentinelOne just dropped a new benchmark that puts frontier AI models through a brutal, real-world test: reverse-engineering a piece of malware tied to Iran’s nuclear program. The results? Most models flunked.
The benchmark, built by SentinelLabs, uses the Fast16 malware — a 2005 Windows threat designed to interfere with LS-DYNA, engineering software allegedly used in Iran’s nuclear weapons development. Think Stuxnet, but earlier. And possibly American-made.
This isn’t another abstract AI quiz. It’s a long-horizon investigation that mimics what human reverse engineers actually do. The goal: see which models can sustain a trustworthy analysis across eight escalating stages, where new evidence repeatedly contradicts their earlier conclusions.
Which Models Passed — and Which Crashed
SentinelLabs tested OpenAI’s GPT-5.5 and the newer GPT-5.6 Sol, Z.ai’s GLM-5.2, and Anthropic’s Opus 4.x. Only GPT-5.6 Sol completed all eight stages — and it needed three separate runs at different reasoning-effort settings to do it.
GPT-5.5 never got past the first stage. The Opus models (4.7 and 4.8) produced solid local analysis but kept declaring the work finished before defects were resolved. GLM-5.2 stalled somewhere in between.
The gap wasn’t about technical skill or raw insight. SentinelLabs calls it project-scale recovery — the ability to withdraw a disproven conclusion, trace everything downstream that depended on it, fix the root cause, and carry that correction through the rest of the investigation. Patching the immediate error isn’t enough. Most models just can’t do that.
Why Fast16 Is the Perfect Test Case
Fast16 is a nasty piece of work. Discovered by SentinelLabs in April, it’s a Windows malware from 2005 that targets LS-DYNA, engineering software used to simulate complex physics — the kind of thing you’d need to design a nuclear weapon. The malware predates Stuxnet, and researchers believe it may have been developed by the United States to sabotage Iran’s nuclear program.
Using a real, historically significant malware sample makes the benchmark brutally practical. It’s not a synthetic puzzle. It’s the kind of investigation that could actually matter in a national security context.
The Human Factor: Why Analysts Still Matter
Even the winner made significant technical mistakes. SentinelLabs researchers concluded that human oversight remains essential — even GPT-5.6 Sol, the only model to finish, wasn’t exactly flawless.
“Senior reverse engineers remain essential,” the researchers said. “Even the strongest runs made semantic errors, accepted weak quality controls, and claimed readiness prematurely. We assess the best current use as supervised investigative agency, with human analysts defining objectives, exposing blind spots, and retaining final publication authority.”
In other words: AI can be a powerful assistant, but it’s not ready to run a malware investigation on its own. Not even close.
What This Means for Security Teams
This benchmark has real implications for how security teams should think about AI tools. If you’re considering using a frontier model to assist with malware analysis, you need to know its limits.
- GPT-5.6 Sol is the only model tested that can sustain a full investigation — but it still needs human oversight.
- Opus models are good at local analysis but tend to stop early, declaring victory before the job’s done.
- GPT-5.5 can’t even get past the initial stage, which is a stark reminder that newer isn’t always better.
For security teams, the takeaway is clear: use AI to augment your analysts, not replace them. Define the objectives, expose blind spots, and keep final authority in human hands. The technology is improving, but it’s not there yet.
This isn’t just about malware analysis either. The same principles apply to other AI-driven security tasks, like vulnerability management automation or AI-powered threat hunting. The models can help, but they need supervision.
The Bottom Line
SentinelOne’s benchmark is a wake-up call for anyone who thinks frontier AI models are ready to handle complex security investigations on their own. They’re not. The technology is impressive — GPT-5.6 Sol’s ability to complete all eight stages is genuinely remarkable — but it’s still a tool, not a replacement for human expertise.
As AI continues to evolve, benchmarks like this will become increasingly important. They give us a realistic picture of what these models can and can’t do, and they help security teams make informed decisions about where to deploy AI assistance. For now, the message is simple: keep your senior reverse engineers close, and treat AI as a powerful ally — not a substitute.
