Results

Leaderboard

Evaluated a model? Add your results to the leaderboard →

Overview

How It Works

The Vulnerability Localization Benchmark (VLoc Bench) is a two-phase agentic evaluation for measuring a model's ability to identify security vulnerabilities in source code.

Phase A Localization

Models are given a repository containing a known vulnerability along with generic CWE descriptions, and must explore the codebase through sandboxed terminal access to identify which files contain the vulnerability. Models are scored on File F1 against the ground-truth patch.

Phase B Verification

Models explore the patched snapshot of the same repo and must confirm the fix holds — without false alarms. Scored on True Negative Rate.

Dataset

Example Vulnerabilities

Browse real vulnerabilities from the benchmark dataset.

Composition

Dataset Distribution

The benchmark contains 500 tasks drawn from 290 repositories, sourced from GitHub Security Advisories across 6 package ecosystems (npm, pip, Maven, Go, Rust, Composer) and 147 unique CWE categories. Each task pairs a repository snapshot containing a known vulnerability with ground-truth file labels derived from the security patch.

Top values by count

Severity

Ecosystem

Language

Top CWE Categories

Analysis

Key Findings

01
Vulnerability localization remains challenging.
Even the best-performing model (GPT-5.5) achieves only 0.229 File F1, indicating significant room for improvement in automated vulnerability detection.
02
Nearly 2 in 5 entries defeat every model.
190 of 500 tasks remain unsolved by all evaluated systems — a structural floor that current approaches cannot breach regardless of model scale.
38%
190 of 500 tasks unsolved by every model
03
Repository size dominates difficulty.
Performance drops 13× from the smallest repositories to the largest. Structural complexity, not vulnerability type, is the primary difficulty predictor.
04
Specialization beats scale.
Purpose-trained 3B models (Antares family) match or exceed 100B+ parameter general-purpose systems — targeted training is high-leverage on security tasks.