Antares in the Wild: Testing on Two New Cybersecurity Benchmarks
A week ago we released Antares, a family of small, open-weight models for agentic vulnerability localization. Given a CWE description and read-only terminal access to a repository, the model searches for the files that contain the bug. On VLoc Bench (our 500-task, repo-scale benchmark) Antares-3B lands near GPT-5.5 while running locally at a fraction of the cost.
A single benchmark only tells you so much, though. The question worth answering is whether that performance holds on tasks built for other purposes. So we evaluated the released models, without modification, on two public benchmarks: CyberGym-localization and a community rebuild of Devin's security evals. Same weights, a terminal, a different distribution.
Why test off-distribution
VLoc Bench is a hard, repo-scale test and Antares performs well on it. But the strongest evidence for a capability is watching it hold up on setups we didn't design. So we evaluated on exactly that: external benchmarks with their own tasks and their own scoring, to see whether the search-and-verify behavior transfers. Two fit the bill:
- CyberGym (UC Berkeley, Dawn Song et al.) is a large-scale cyber-capability benchmark: ~1,500 real vulnerabilities pulled from 188 open-source projects via OSS-Fuzz, where the headline task is generating a working proof-of-concept exploit.
- Devin's Security Swarm eval (Cognition) is 50 real, recent CVEs across 14 languages, graded on whether the tool finds the one known bug. Cognition never released it, so we use Matt Rickard's open rebuild of 34 of the 50 fixtures.
CyberGym-localization
CyberGym's headline task is writing a working proof-of-concept exploit. We don't evaluate that. We evaluate the localization step: given the vulnerability description and the crash, find the file that's actually broken. You can't exploit what you can't find, and localization is the cheap front half of the pipeline, which is exactly the part a small model can own.
Across all of CyberGym, Antares-3B reaches 73.7% recall for about $1.60 a run, sitting in the same band as hosted models that cost hundreds. The released models hold up too: 1B clears GPT-OSS-120B and GPT-4o, and 350M matches GPT-5-nano for roughly a dollar.
Because the open models run locally, localization-per-dollar for Antares sits well above every hosted option here, and the source code never leaves the machine.
The Devin security-eval rebuild
Our second test uses a community rebuild of Devin's security evals. Cognition never released theirs, so exact replication isn't possible. Matt Rickard rebuilt 34 of the 50 fixtures in the open, and we evaluated the released Antares models against it, scored with recall (the same grade the original used).
In the same recall band as a ~$90/run hosted agent, the released models run for under a dollar and stay local. Roughly two orders of magnitude cheaper per run.
This is the public 34/50 rebuild, not Cognition's exact set, and Security Swarm is a hosted product. The Antares numbers (marked *) are approximate, extrapolated from the rebuild. Read it as directional, not a head-to-head. And none of it runs without the open rebuild, so credit to Matt Rickard for making it reproducible.
What this means for small security models
Two more benchmarks, and localization holds up on both. The search-and-verify behavior generalizes: it transfers to new repositories, new harnesses, and new grading rules.
And it does so cheaply. Localization shouldn't require a hosted API call per commit, nor should it ship source code to someone else's cloud. These models are small enough to run locally, cheap enough to gate every push, and (on these numbers) capable enough to justify it.
Try it
- Open weights (Antares-350M, Antares-1B): huggingface.co/collections/fdtn-ai/antares
- VLoc Bench: github.com/cisco-foundation-ai/vulnerability-localization-benchmark
One note: Antares-3B is not yet released, but it is coming soon. Only 350M and 1B are open-weight today. We include 3B above to show where the family is headed.