New Cybersecurity Reasoning Benchmarks in FAITH: CTI-Reasoning and CWE-Prediction
Most cybersecurity benchmarks test what a model has memorized — definitions, known CVEs, textbook facts. But real security work demands reasoning: connecting attack patterns, weighing evidence, and classifying threats no one has labeled yet. FAITH (Foundation AI's Testing Hub) now includes two new benchmarks built to measure exactly that.
CTI-Reasoning and CWE-Prediction evaluate the analytical tasks security practitioners face daily: reasoning through relationships in threat-intelligence taxonomies, and classifying vulnerabilities from unstructured, real-world advisories.
The Benchmarks
CTI-Reasoning
This benchmark tests multi-hop logical analysis and comprehension of complex relationships within MITRE's cybersecurity frameworks. Unlike traditional multiple-choice tests that focus on memorization, 96% of questions require analytical reasoning rather than factual recall — specifically, 77.5% demand analysis-based cognitive processing and 22.5% require comprehension-based reasoning.
Questions are derived from CAPEC (Common Attack Pattern Enumeration and Classification) and CWE (Common Weakness Enumeration) documentation, focusing on understanding threat patterns, attack techniques, and weakness relationships.
Here's a real example — watch a reasoning model work through it:
A. Failures in electromagnetic protection allow signal-based attacks without leaving physical traces. B. Physical security can be bypassed, implying core MDFI techniques don't require device access. C. Attack depends entirely on security failures unrelated to physical device access. D. Strong physical security inherently allows MDFI attacks without further protection.
Models must output their answer in the format 'Answer: $LETTER' on the last line of their response, enabling automated evaluation via regular expression extraction.
Dataset: fdtn-ai/cti-reasoning
CWE-Prediction
This benchmark evaluates vulnerability classification from real-world security advisories. Given a CVE or GitHub Security Advisory description, models must predict the appropriate CWE identifier from 263 possible weakness types.
Question: What Common Weakness Enumeration (CWE) identifier is associated with
CVE-2025-26949, which describes an 'Improper Neutralization of Input During Web
Page Generation (Cross-site Scripting)' vulnerability in bPlugins Team Section
Block that allows Stored XSS?
Final Answer: CWE-79Models are required to output the CWE ID in the format CWE ID: CWE-$id on the last line, with automated extraction handling both proper format (CWE-79) and variations (CWE: 79, id: 79).
The benchmark is constructed identically to CTIBench-RCM (vulnerability description to CWE mapping), but uses recent data from 2025 CVE entries and 2024-2025 GHSA advisories. This design tests whether models can generalize the same vulnerability classification capability to previously unseen, temporally-shifted data — ensuring the data is new to the model and measuring real-world applicability rather than memorization.
Why does this matter? Most models' training data predates the benchmark's 2024–2025 sources, so they can't have memorized these mappings — they have to reason. The timeline below plots each model's training-data cutoff against the benchmark's data window:
Dataset: fdtn-ai/cwe-prediction-benchmark
Using the Benchmarks
Both benchmarks are available in FAITH's latest release (0.4.0) and can be evaluated like any other benchmark:
# Evaluate a single benchmark
faith run-all \
--model-paths fdtn-ai/Foundation-Sec-8B-Instruct \
--benchmarks cti-reasoning \
--num-trials 5
# Evaluate both benchmarks
faith run-all \
--model-paths fdtn-ai/Foundation-Sec-8B-Instruct \
--benchmarks cti-reasoning,cwe-prediction \
--num-trials 5Dataset Access
Both datasets are available on HuggingFace and can be loaded directly:
from datasets import load_dataset
# Load CTI-Reasoning (200-question test split)
cti = load_dataset('fdtn-ai/cti-reasoning', split='cti_reasoning_test')
# Load CWE-Prediction
cwe = load_dataset('fdtn-ai/cwe-prediction-benchmark', split='cwe_prediction_benchmark')How Models Perform
Both benchmarks are challenging and unsaturated — even frontier models leave substantial headroom, which makes them useful yardsticks for measuring progress. The scores below span a range of open and proprietary models, from the Foundation-Sec-8B-Reasoning technical report (average over 5 trials).
View full results table
| Model | CTI-Reasoning | CWE-Prediction |
|---|
Full results, including standard deviations and the broader cybersecurity benchmark suite, are available in the technical report.
License & Attribution
Both benchmarks are derived from publicly available sources with proper attribution:
- CTI-Reasoning: CAPEC and CWE (The MITRE Corporation) — freely available with attribution
- CWE-Prediction: CVE/NVD (NIST, public domain) and GHSA (GitHub, CC-BY 4.0)
When using these benchmarks, please cite the original data sources. Full license information is available in the benchmark metadata and dataset cards on HuggingFace.
What's Next
Static benchmarks lose signal as models train on them. By pairing reasoning-intensive questions with vulnerability data that postdates most training cutoffs, CTI-Reasoning and CWE-Prediction resist that decay. We hope they give the community a durable way to measure genuine security reasoning, not memorization.
Get started:
- FAITH on GitHub
- CTI-Reasoning dataset
- CWE-Prediction dataset
- Foundation-Sec-8B-Reasoning technical report
FAITH is open source. If you're using these benchmarks and have feedback on the evaluation prompts, dataset quality, or scoring methodology, we'd like to hear it. If you develop additional cybersecurity reasoning benchmarks or evaluation techniques that generalize beyond your specific use case, consider contributing them back.