garak

garak is a free, open-source command-line scanner for probing language models and the systems built around them. It best fits security-minded developers and evaluation teams that need configurable checks they can run against hosted or self-hosted models. The main trade-off is its deliberately text-only scope, so it is not a complete assessment for multimodal or non-text behavior.

Overview

garak treats model security as an adversarial exercise. Its probes try to make a model fail, while detectors inspect the resulting output for patterns such as unsafe content or a failure to refuse. This gives teams a repeatable way to look for weaknesses rather than relying only on manual review.

The stated threat coverage includes hallucination, data leakage, prompt injection, misinformation, toxicity generation, and jailbreaks. That range makes garak a practical first-pass scanner across several common LLM risk areas. It remains a focused security tool, however, rather than a general dashboard for every aspect of model quality.

Key features

A standard scan can run without supervision and produce a report, while configuration can be customized when a generic scan does not match your system or threat model. This balance suits recurring checks in development and security workflows, although meaningful results still depend on choosing probes and detectors appropriate to the model and use case.

Integrations cover OpenAI, Hugging Face, Cohere, Replicate, AWS Bedrock, LiteLLM, REST-accessible models, and custom Python integrations. You can also run Hugging Face models locally or reach them through an inference API or private endpoint. That breadth is useful when a team needs one probing approach across different providers, but each integration still brings its own access and deployment considerations.

Output is split across screen output, a detailed report log, a hit log, and a debug log. The separation helps distinguish findings from diagnostic information when a scan needs investigation or repeat runs. garak also supports custom metrics, LLM-as-a-judge assessments, and safety evaluations, extending it beyond a single fixed scoring method.

Pricing

There is no paid tier or free trial to weigh against the free plan. The zero-cost, open-source model is a strong fit for teams able to operate a command-line scanner themselves, while organizations seeking hosted service management or commercial support arrangements should look elsewhere.

Platforms

garak runs on Linux, macOS, and Windows, and supports self-hosted deployment. It was developed as a Linux and macOS command-line tool and can be installed with pip or from source. Self-hosting keeps model and scan control in your environment, which suits privacy-sensitive evaluation, but it also puts installation and operation on your team.

Who it's for

Choose garak if you are assessing LLM security across several providers, local models, or private endpoints and want configurable automated probing at no software cost. It is particularly well matched to engineers and red teams comfortable with command-line workflows, logs, and custom Python integrations.

It is a weaker fit when your review must include image, audio, or other non-text outputs, or when you want a hosted interface rather than a self-managed tool. The reference scope covers mainstream any-to-any large models but assesses text output only.

Pros and cons

• Pro: Broad provider support lets one scanner cover OpenAI, Hugging Face, Cohere, Replicate, AWS Bedrock, LiteLLM, REST models, and custom integrations.
• Pro: Automated scans and configurable probes make repeatable security checks practical.
• Pro: Separate report, hit, screen, and debug outputs support both review and troubleshooting.
• Pro: Free, open-source, self-hosted deployment avoids subscription cost and keeps operation under your control.
• Con: Text-only assessment leaves multimodal and other non-text behavior outside its stated scope.
• Con: A command-line, self-hosted workflow requires technical ownership of installation, configuration, and execution.

Alternatives

AI Red Teaming Tools, LLM Evaluation Tools, and AI Security Testing Tools are useful starting points for comparing categories and neighboring products.

AgentSeal is the better choice if you want a free tier with 30 basic probes and an MCP registry, while its Pro plan adds 311 probes, OAuth runtime MCP scanning, and PDF export. Pick OpenSecureAI Scanner when a hosted service with a $49-per-month Pro plan, 50,000 API requests per month, and hosted remediation is more important than self-hosting. Promptfoo suits teams wanting a free Community plan with 10,000 red-team probes per month and broad evaluation integrations. RedAmon is an open-source, MIT-licensed Docker option for commercial or personal self-hosting.

F5 BIG-IP APM is a paid, enterprise deployment option available across virtual, hardware, or hybrid environments. Giskard is worth choosing for its free local library with adversarial LLM vulnerability scans and RAG evaluation. NVADER is a web alternative with a free plan for one app and app scanning. ProofLayer fits teams seeking a free MIT-licensed CLI and MCP scanner with 1,700-plus detection rules and MCP tool testing.

Verdict

garak is a strong choice for technical teams that want free, configurable, automated LLM security probes across hosted and local models. Its provider coverage, logs, custom metrics, and self-hosted operation give it real value for repeatable text-based assessments. Look elsewhere if you need multimodal coverage or a managed, graphical security platform.