CyberSecEval
CyberSecEval 4 is a free benchmark suite for assessing cybersecurity vulnerabilities and defensive capabilities in large language models.
What CyberSecEval is
CyberSecEval 4 is a free benchmark suite for assessing cybersecurity vulnerabilities and defensive capabilities in large language models. Its tests examine responses to cyberattack requests, incorrect refusals of borderline benign queries, and insecure code suggestions in both instruction and autocomplete settings. Prompt-injection tests cover textual, multilingual textual, and visual attacks, with a judge LLM assessing whether injected instructions succeeded. A code-interpreter benchmark evaluates compliance with malicious prompts and classifies judged responses by severity. AutoPatchBench measures patch-generation agents on C and C++ vulnerabilities found through fuzzing, while CyberSOCEval, developed with CrowdStrike, covers malware analysis and threat-intelligence reasoning. The suite supports API, Linux, and self-hosted use. The guide names API support for OpenAI, Anyscale, and Together, and explains how to add custom providers for self-hosted models. Setup involves a Python virtual environment and command-line benchmark runs. The benchmarks are licensed under MIT for research and commercial use. AutoPatch has significant compute and storage requirements, and Apple Silicon MacBooks are not currently supported.
Good
- Free, open-source benchmark suite.
- Includes textual, multilingual, and visual prompt-injection tests.
- Covers insecure code suggestions and malicious code-interpreter prompts.
- Supports API, Linux, and self-hosted use.
- MIT license permits research and commercial use.
Watch out
- AutoPatch recommends at least 80 CPUs and 8 TB storage.
- Apple Silicon MacBooks are not supported for AutoPatch.
Who it suits
CyberSecEval suits developers and researchers evaluating cybersecurity risks or defensive capabilities in generative AI models. Teams considering AutoPatch should account for its substantial compute and storage requirements.
Verdict
CyberSecEval covers several distinct ways models can create or resist cybersecurity risks, including patch generation and defensive SOC tasks. Its AutoPatch requirements are demanding, so assess the available compute and storage before including that benchmark.
Plans and prices
From CyberSecEval's own pricing page, checked 8 Oct 2026.
| Plan | Price | What it gives you |
|---|---|---|
| CyberSecEval 4 | Free | Open source benchmark suite |
The facts
Each one read from CyberSecEval's own pages, with the date it was checked.
What it does
- Automated evaluation
- The original research describes automated test-case generation and an evaluation pipeline for LLM cybersecurity safety.checked 15 Oct 2026
- AutoPatchBench
- AutoPatchBench measures AI patch-generation agents on C/C++ vulnerabilities found by fuzzing.checked 8 Oct 2026
- AutoPatch requirements
- AutoPatch requires Podman and substantial compute and storage; the page recommends at least 80 CPUs and 8 TB storage and says Apple Silicon MacBooks are not currently supported.checked 8 Oct 2026
- Code interpreter
- The code interpreter benchmark assesses whether models comply with malicious prompts and categorizes judged responses as extremely malicious, potentially malicious, or non-malicious.checked 8 Oct 2026
- Current version
- CyberSecEval 4 adds AutoPatchBench for measuring an LLM agent’s ability to automatically patch vulnerabilities in native code.checked 15 Oct 2026
- CyberSecEval 2 additions
- CyberSecEval 2 added prompt-injection, code-interpreter-abuse and vulnerability-exploitation evaluations.checked 15 Oct 2026
- CyberSecEval 3 coverage
- CyberSecEval 3 evaluates eight risks across risks to third parties and risks to application developers or end users.checked 15 Oct 2026
- CyberSecEval 3 features
- CyberSecEval 3 added automated social engineering, scaled manual offensive operations and autonomous offensive operations tests.checked 15 Oct 2026
- Documentation depth
- The repository documents benchmark commands, provider formats, self-hosted integration, datasets, judging stages and developer tests.checked 15 Oct 2026
- Extensibility
- Developers can add providers by implementing an LLM class, registering it in the factory and supplying query and chat methods.checked 15 Oct 2026
- False-refusal metric
- CyberSecEval 2 introduced False Refusal Rate to quantify the tradeoff between rejecting unsafe requests and answering benign ones.checked 15 Oct 2026
- Independent assessment
- The peer-reviewed critique describes CyberSecEval as useful in a young field but identifies features that limit its utility.checked 15 Oct 2026
- Independent implementation limit
- Inspect Evals intentionally omits the autonomous-uplift and autopatching prototypes from its public CyberSecEval 4 task surface.checked 15 Oct 2026
- Installation
- The documented setup creates a Python virtual environment and installs dependencies from requirements.txt.checked 15 Oct 2026
- Intended users
- The project describes itself as tools and evaluations intended to help the community build responsibly with open generative AI models.checked 8 Oct 2026
- License
- Cybersecurity evaluations and benchmarks in Purple Llama are licensed under MIT, while included Llama safeguard models use separate Llama Community licenses.checked 15 Oct 2026
- Limit: narrow ruleset
- The critique reports that Meta’s 89 Semgrep rules cover eight languages, versus 2,116 rules and 28 languages in the comparison repository.checked 15 Oct 2026
- Limit: prompt construction
- The critique found 23.5% of Instruct prompts problematic because they tested refusal behavior rather than genuine insecure-code propensity.checked 15 Oct 2026
- Limit: static analysis
- An independent critique says CyberSecEval’s insecure-code detector relies on 189 rules and cannot understand code context, creating false positives.checked 15 Oct 2026
The company
- Company founded
- Meta’s SEC filing says the company was incorporated in Delaware in July 2004.checked 15 Oct 2026
- Company headquarters
- Meta lists its principal executive offices at 1 Meta Way, Menlo Park, California 94025.checked 15 Oct 2026
- Company legal name
- Meta’s SEC filing identifies the maker as Meta Platforms, Inc., a Delaware corporation.checked 15 Oct 2026
- Company size
- LinkedIn lists Meta as having more than 10,001 employees and headquarters in Menlo Park, California.checked 15 Oct 2026
Money
- Independent cost observation
- An Inspect Evals report estimated full one-epoch CyberSecEval 4 runs at roughly $29 to $213 per model depending on the model used.checked 15 Oct 2026
Alternatives to CyberSecEval
- 1
PromptGuard
Free planWebAPIExtensionLinux7.9From $19 a month - 2
NVIDIA NeMo Guardrails
Free planOpen sourceAPILinuxmacOSSelf-hosted7.4Free plan - 3
Oracle Mobile Authenticator
Free planAndroidiOSWindows6.9Free plan - 4
Enkrypt AI Guardrails
Free planWebAPISelf-hosted6.8From $1,608 a year - 5
OpenSecureAI Prompt Firewall
Free planWebSelf-hosted6.7From $49 a month - 6
Google Cloud Secret Manager
Free planWeb6.5From $0.02 a month