VLoc Bench: Can Agents Find Vulnerable Code at Repository Scale?

When security engineers investigate a vulnerability, one of their first questions is simple: which files contain it?

Most cybersecurity benchmarks skip this step. They give models the relevant code or tell them where the vulnerability is, then test whether they can detect, repair, or exploit it.

Today, we're releasing the Vulnerability Localization Benchmark (VLoc Bench) to test whether agents can find vulnerable files on their own. Given only a weakness class and read-only access to a repository, an agent must identify the affected files, or, once the repository has been patched, correctly report that the weakness is no longer present.

Our results show that vulnerability localization is far from solved. The strongest system achieves just 0.229 File F1, and on 38.4% of tasks, none of the models find a single correct file. We also find a key tradeoff: the systems best at finding vulnerabilities are not necessarily the best at recognizing when a repository has already been patched.

Inside VLoc Bench

Every task is built from a GitHub Security Advisory, a public record linking a CWE class to a repository and the commit that fixed it. From that commit we take two snapshots: the pre-push state, which still contains the vulnerability, and the post-push state, which contains the fix. Ground-truth labels come from the patch diff deterministically: every implementation file the fix touched, minus tests, docs, and config. What is left is 500 tasks from advisories disclosed between 2016 and 2026.

The model never sees the advisory. It receives the generic MITRE description of one weakness class, with no advisory text, no CVE identifier, no fixing commit, no file hint, and no line range. Because a weakness class says what kind of vulnerability to look for but nothing about where it lives, every file the agent names has to come out of its own search.

Two phases, one interface

The paired snapshots let each task be scored twice, under an identical prompt, toolset, and command budget. Only the repository state changes.

$ ls src/
signed-xml.ts  utils.ts  validator.ts

$ grep -r "verifySignature" src/
src/signed-xml.ts: verifySignature(cert) {

$ cat src/signed-xml.ts | head -30
export class SignedXml {
  verifySignature(cert) {
    // ⚠️ Only checks signature validity
    // Missing: authorization check
    return this.validateSignature(cert);
  }
}

> submit_vulnerable_files
["src/signed-xml.ts"]
✓ Correct! F1 = 1.0

$ cat src/signed-xml.ts | head -30
export class SignedXml {
  verifySignature(cert) {
    // ✓ Now checks signature validity
    return this.validateSignature(cert);
  }
}

> submit_vulnerable_files
No Vulnerability

The interface and the metrics

Every system runs against the same read-only terminal (file listing, text search, file viewing, metadata queries) in a fresh, network-disabled container. The agent ends a task by calling submit_vulnerable_files with a ranked list of paths, or submit_no_vulnerability_found. It gets 15 commands and 20 turns, and every score is the mean of three runs.

What is in the 500 tasks

The distribution follows the advisory database rather than a balanced sample, so Go dominates and Composer contributes a single task. Repositories range from 12 KB to 840 MB; the median task has three ground-truth files.

The Leaderboard

We evaluated 27 language models, plus four static-analysis tools as baselines, all under the same prompt, tools, and budget. The models span frontier closed-source systems, open-weight models from 350M to 753B parameters, and models trained specifically for code search or vulnerability localization.

Parameter count is a weak predictor

Larger systems often underperform smaller ones in both open-weight and closed-source families. Llama-3.3-70B scores 0.012 while the 2B Gemma-4-E2B scores 0.039, Qwen3.5-122B ties Qwen3.5-27B at 0.091, and GPT-5 Mini beats GPT-5, which is consistent with GPT-5's system card reporting greater conservatism on dual-use cybersecurity tasks. General software-engineering capability is not a reliable proxy here: a system can underperform because safety alignment constrains dual-use work, or because security reasoning is out of distribution relative to its coding training.

Task-specific training is the sharpest illustration. Antares-3B, our own 3B model trained for agentic vulnerability localization, reaches 0.223 File F1, second overall and ahead of GPT-5.5 at default reasoning effort. Antares-350M scores 0.135 at 350M parameters, above every general-purpose open-weight model except the 753B GLM-5.2. Fitted across the general-purpose open-weight models alone, the parameter-count trend is positive but weak (r=0.50), and the specialized models sit well off it.

Static analyzers are a real baseline

Semgrep at 0.086 File F1 outperforms most of the open-weight models we tested, including two above 100B parameters. General-purpose models under 20B do worse still: CodeScout-14B, the best of them at 0.044, scores about half what Semgrep does. The systems that clear Semgrep by a wide margin are either frontier models or trained for this task.

Finding Vulnerabilities Is Not the Same as Knowing When Not To

Phase B reverses the question. The repository has been patched; the recorded vulnerability is gone; the correct answer is to report no file. Ranking by true negative rate produces a substantially different ordering than Phase A, which is the most consequential finding in the release.

The contrast shows up even within one family. GPT-5.5 (xhigh) leads Phase A at 0.229 File F1 but reports files on 72% of already-patched repositories. GPT-5 Nano is the mirror image: 0.024 File F1, but it correctly clears 86.8% of patched repositories.

The Granite base models make the caveat explicit. Granite-4.0-1B posts a perfect 1.000 true negative rate and 0.000 File F1: it never submits a file anywhere, so it is right by default on patched code and wrong everywhere else. Reading either phase alone rewards a degenerate strategy: score only localization and you favor systems that report broadly; score only patched repositories and you favor systems that never commit.

This matters beyond the leaderboard. Alert fatigue is a persistent operational problem, so localization that improves without preserving the ability to recognize already-patched code is not something a security team can use. False-positive control belongs in the definition of capability, not in a footnote.

What Makes Localization Hard

Aggregate scores say how well systems do, not why tasks are hard. To separate the two, we fit a Lasso regression predicting per-task File F1 from 52 repository and metadata features available before any model runs.

Repository structure carries 4.5× the predictive weight of model identity, and a model-only regression explains almost nothing about which tasks get solved. The strongest single predictor is code concentration: repositories whose lines sit in a few large files are easier. Larger repositories, deeper trees, and Go projects score lower, while the number of ground-truth files carries negligible weight. How the code is organized matters more than how many files you have to find.

Scale and ecosystem interact, and the interaction is clearest in the unsolved set. In 38.4% of tasks, every model scores File F1 = 0; large Go and Maven repositories account for 47% and 31% of that subset, and Go alone is 43% of the benchmark. Adding models or parameters does not dissolve it.

How Runs Fail

File F1 collapses very different behaviors into one number: a run that stops after two commands and a run that submits a dozen wrong files both score 0. So we label every non-perfect Phase A trial, about 38,500 of them, from its trace, using command counts, precision, recall, and submission status rather than a model judge.

Two categories dominate. Exhausted budget (32.3%) is a run that issues at least 13 of its 15 commands and never submits: search that does not converge within the horizon. Wrong files (27.4%) is the opposite: a confident submission with no overlap with the ground truth.

Premature termination (16.9%), stopping after at most three commands, is concentrated among smaller systems. Partial recall (8.5%) is the most encouraging failure and more common among stronger systems: the run finds at least one relevant file but misses others, so multi-file localization stays hard even when the search reaches the right code.

Across models, mean commands per task correlates with mean File F1 at r=0.72 (p<0.001): weaker systems stop early, stronger ones keep searching. Higher performers also spend more of that budget on targeted search and file reading (rg, grep, cat) than on directory listing, and reliability gates everything: tool-error rates above 15% coincide with lower scores, because failed commands consume budget without yielding evidence.

Run It Yourself

The benchmark, the harness, and the analysis code are public. All 500 tasks download from public GitHub repositories at the recorded commit SHAs, with no gated access, and each is verified against a deterministic content hash, so a snapshot either matches the manifest or fails loudly.

git clone https://github.com/cisco-foundation-ai/vulnerability-localization-benchmark.git
cd vulnerability-localization-benchmark
pip install -e .

# Build the sandbox image the agent runs inside
docker build -t vulnerability-localization-benchmark-sandbox .

# Fetch and verify all 500 paired repository snapshots
python data/downloader_and_verifier.py --source-dir /path/to/dataset/

Then evaluate each phase: Phase A on the vulnerable snapshots, Phase B on the patched ones.

# Phase A - localization on vulnerable repositories
PYTHONPATH=src python3 -m vulnerability_localization_benchmark.cli \
    --config configs/default.yaml --api-base http://localhost:8200/v1 \
    --model-name your-model --runner vllm --output-dir results/my-run \
    --phases a --workers 16

# Phase B - verification on patched repositories
PYTHONPATH=src python3 -m vulnerability_localization_benchmark.cli \
    --config configs/default.yaml --api-base http://localhost:8200/v1 \
    --model-name your-model --runner vllm --output-dir results/my-run \
    --phases b --workers 16

What a full sweep costs

A full 500-task Phase A sweep spans a 170× cost range, from $0.60 for Antares-350M on a single H100 to $141 for GPT-5.5 xhigh through the OpenAI API. Open-weight models up to 31B parameters, served with 16 workers on one H100, finish in under an hour, cheap enough to sit inside a CI stage.

Cost is also why some models are missing: a standardized sweep would exceed $600 at Claude Opus 4.8 pricing, and an unconstrained agentic configuration using Claude Code with subagent spawning costs $1,658 for the same 500 tasks. We will add these as budget permits.

What's Next

Because the task distribution is fixed, VLoc Bench works as a system-level evaluation, not just a model ranking. Hold the 500 tasks constant and swap harnesses, prompts, search strategies, and tool orchestration. That separates model capability from the surrounding scaffolding.

Future versions will expand coverage with held-out and post-training-cutoff vulnerabilities, additional ecosystems, and improved validation of patch-derived labels.

Get started:

• VLoc Bench on GitHub: manifest, snapshot tooling, evaluation harness, scoring, and analysis scripts
• Leaderboard: filter by model family and type, and submit your own results
• Technical report: full methodology, regressions, and per-model failure profiles

Labels come straight from patch diffs, which makes them reproducible but not perfect. If you find a task that looks mislabeled, a scoring case we did not anticipate, or a harness assumption that breaks your agent, open an issue on GitHub.