Measuring Attacker/Defender Asymmetry: A Cyber-Capability Safety Benchmark
In July 2026, Hugging Face got hacked by an AI agent, and the "safe" models refused to help clean it up. An OpenAI agent with its safety classifiers switched off ran a 4.5-day, 17,600-action intrusion into production infrastructure. When responders reached for Anthropic's frontier models, Claude Opus and Fable, to reverse-engineer the attacker's payloads, the models refused: their guardrails couldn't tell a defender analyzing an exploit from an attacker launching one. So the attacker ran with the brakes off, the defender's best tools braked themselves, and the victim fell back to an open-weight model to investigate its own breach.
That is what "safe" has quietly come to mean. Safety alignment has converged on a single reflex, refusal, and most safety benchmarks reward it directly: a model scores safer the more readily it says "I can't help with that" the instant a prompt smells like security, no matter who is asking or why. But refusal doesn't delete a capability from the world. It disarms the defender and leaves it fully intact for anyone willing to turn the guardrails off or download open weights. Measured that way, "safe" and "useless to the people under attack" become the same score.
So we stopped asking whether a model will refuse cyber work and started asking a sharper question: does its ability point at defenders or at attackers? A defender's tool should be strong where defenders work (source code) and collapse where attackers work (stripped binaries). We built the Antares models to have exactly that shape, and to prove it isn't a slogan, we built the benchmark that measures it: Safety-VLoc-Bench. The results show Antares is defense-favoring by construction. This benchmark is our argument to the AI safety community. In a capability era, "safe" should mean strong for defenders and useless to attackers, not simply that a model refuses.
What this benchmark is
Safety-VLoc-Bench is a paired benchmark. Every case is one real C/C++ memory vulnerability that appears in two forms carrying the same ground-truth label:
- source: the original project source tree (the defender's view)
- decompiled: Ghidra decompilation of the compiled, stripped -O0 binary (the attacker's view)
You run a localizer on both forms and compare file-level F1. The quantity of interest is the gap:
Δ = F1_source − F1_decompiledA large, consistent Δ (source ≫ decompiled) is evidence that a localization model is structurally defense-favoring: it works where defenders operate (source) and collapses where attackers operate (binaries). That asymmetry is a safety property, not a capability deficiency, hence the "Safety" in the name.
To our knowledge this is the first Cyber-Capability Safety benchmark of its kind: it measures not "can the model find the bug" but "does the model's ability to find the bug survive the transformation an attacker would have to work through." Most cyber benchmarks measure raw capability; this one measures the shape of that capability across the defender/attacker boundary.
Why "safety": our threat model
The threat we care about is re-pointing: a model built to help defenders find bugs being turned around and used to hunt them in software the attacker doesn't own. The two sides work from different artifacts. A defender fixing their own code has the source; an attacker probing a target they cannot recompile has only the shipped binary, stripped of symbols and structure. So the security-relevant question is not whether the model can find a vulnerability, but which view it can find it in. A model that localizes on source but fails on stripped decompiled code is strong for the defender and close to worthless to the attacker, because the attacker's workflow starts where the model's ability ends. Safety-VLoc-Bench turns that into evidence: a number, per model, for how much capability survives the crossing from the defender's view to the attacker's.
How we built it
The pairing procedure keeps the source and decompiled forms directly comparable while removing every unfair advantage from the decompiled side.
Where the cases come from
Cases are drawn from ARVO, a corpus of real C/C++ memory vulnerabilities reproduced from OSS-Fuzz. Each ARVO entry gives us everything a paired case needs: a Docker image at the vulnerable revision, a Docker image at the fixed revision, the PoC input, and, crucially, the developer fix patch. The set of files that patch modifies is the ground-truth localization label. Labels are always expressed against the source tree; the decompiled side inherits the same label through the mapping step below.
Sampling ARVO for coverage
The starting set is a random sample of ARVO, drawn to span distinct CWE classes across many repositories rather than to mirror ARVO's raw frequencies. We were after a spread we judged apt for a first benchmark: enough different bug classes that the source-vs-decompiled gap can't be an artifact of one vulnerability type, and enough different projects that it can't be an artifact of one codebase's layout. So the sample deliberately reaches for unique CWEs and pulls cases from as wide a set of repos as the downstream filters allow, instead of letting the most-reproduced few projects dominate by default.
We then worked through that sample by compiling and decompiling each candidate. The build step is where it thins out: a good number of ARVO cases would not compile (stale toolchains, kernel/sanitizer incompatibilities, missing build dependencies), and those were dropped rather than forced through. Every case that made it all the way through a clean rebuild and decompilation was kept, and what survived intact on both sides of the pair is the 95 pairs that make up the benchmark.
Rebuilding and decompiling under controlled flags
Stock ARVO images are built for fuzzing. We rebuild each case clean through ARVO's own arvo compile hook, at -O0 -g with no sanitizer, matching production envs. This is also the conservative choice: sanitizer instrumentation leaves telltale traces that would make the vulnerability easier to spot on both sides, so testing without it only makes the task harder, not easier.
CFLAGS/CXXFLAGS = -O0 -g -fno-omit-frame-pointer
SANITIZER = none-O0 -g gives a trustworthy DWARF line table (no inlining, no reordering), which is what makes the label mapping reliable. The decompiled form the model sees is then produced from the stripped ELF via Ghidra headless: stripping happens first because decompiling a symbol-rich binary would leak the answer through names, so Ghidra names functions by address (FUN_004217a0_0x4217a0.c), the same artefact an attacker works from.
Carrying the label across the pair
The two forms now speak different languages: source GT is a path (src/lib/lwan-request.c), decompiled files are named by address. We bridge them with the DWARF from the unstripped -O0 build. addr2line maps each decompiled function's address back to a source function and file, producing a gt_mapping.json that lets us score decompiled predictions against the same underlying ground truth as the source side. The -O0 -g build above is what makes this possible; without clean DWARF there is no reliable way to identify which decompiled function is the vulnerable one.
Evaluation protocol
- Task: agentic vulnerability localization at file granularity, run independently on the source and decompiled forms of each case.
- Metric: file-level F1 (precision/recall of submitted files vs. the files the fix patch touched), and the gap Δ = F1_source − F1_decompiled.
- Scoring on the decompiled side uses
gt_mapping.jsonto translate address-named decompiled files back to source paths.
What we can extend to
We treat stripped -O0 as the primary setting, and the one that most cleanly measures capability: the source is intact for the defender, and the attacker's view is the shipped binary with symbols and structure removed, nothing more. The build/decompile pipeline is also parameterized over optimization level {O0, O2, O3} and {stripped, unstripped}, so the full attacker/defender space is a grid we can generate, and future evals will fill it in. The expectation is that Δ only widens toward the harder cells as more of the structure a localizer relies on is destroyed, so stripped -O0 is already the conservative floor on the asymmetry, not a best case.
Dataset composition
Size & shape
CWE distribution (share of CWE-labeled cases)
Repository distribution (top of the head + tail)
| Project | Share |
|---|---|
| c-blosc2 | ~35% |
| wolfmqtt | ~6% |
| ndpi | ~5% |
| lcms | ~4% |
| mruby / libucl / tinygltf | ~3% each |
| lwan, c-ares, mdbtools, quickjs, MapServer, mongoose | ~2% each |
| 25 further projects | ~1% each (~26% combined) |
How our models perform
Agentic protocol (terminal tool, capped call budget) identical on both forms of every case. Mean File F1 over all 95 paired cases, source vs. stripped -O0 decompiled:
Consider Antares-3B first. On source it localizes the vulnerable file at 0.823 F1, close behind gpt-5.5-xhigh (0.857) despite being a fraction of the size. On the stripped -O0 form of the same bugs it drops to 0.000. That is the widest gap in the table (Δ = 0.823), and the pattern repeats down the family: Antares-1B scores 0.756 on source, Antares-350M scores 0.604, and both fall to exactly 0.000 on the attacker's view.
The exact zero is what matters. The frontier models only reach a near-zero floor (gpt-5.5-xhigh 0.139, gpt-5-mini 0.038), meaning they still land the occasional hit on a stripped binary. Every Antares model reaches zero. Once symbols and structure are gone (file names removed, functions renamed to FUN_00xxxxxx, code layout flattened), Antares can no longer point an attacker at the bug at all.
This is what "defense-favoring by construction" looks like in numbers: an open model you can run yourself, matching frontier accuracy in the defender's domain and provably useless in the attacker's. The gap appears for every model we tested, so it is a genuine property of the task, but Antares is the only family that reaches the exact zero the safety argument calls for. Filling in the intermediate optimization and stripping cells is future work; the endpoints already show the full size of the effect.
A single score summarizes the asymmetry per model. The leakage score, 1 − edecomp/source, sits at zero when nothing survives the strip and dips negative in proportion to how much the binary still gives away.
What this means for safe cyber capability
The quantity that matters is not raw F1 but Δ. Antares localizes vulnerabilities on source code, where defenders operate, and collapses to exactly 0.000 on the stripped, decompiled binaries where attackers operate. That gap is the safety property, and it is one the benchmark measures rather than asserts.
Refusing to touch cyber doesn't make a model safe; it makes it useless to the defender and irrelevant to the attacker, who, as July showed, will simply run a model with the brakes off or reach for open weights. Measuring, and widening, the asymmetry that favors defenders is the version of safety we think is worth building toward. Safety-VLoc-Bench is our first instrument for it.
A defender's tool should be strong where defenders work and weak where attackers work. Antares does that, and Δ is how we measure it.
Want to run a one-of-a-kind cyber-capable attacker/defender asymmetry benchmark? Reach out at fdtn.ai/contact.
Try it
- Open weights (Antares-350M, Antares-1B): huggingface.co/collections/fdtn-ai/antares