Strands Evals

What Strands Evals is

Strands Evals is an open-source Python SDK and CLI for evaluating AI agent behavior during development and after deployment. It scores outputs and trajectories, helps diagnose failures, probes unsafe behavior, and can simulate users and tools. Evaluators can assess an output, tool call, trace, or entire session, and multiple evaluators can be used in one experiment. Built-in groups cover quality, safety, multimodal responses, agent behavior, and skill selection and instruction following. Deterministic checks such as Equals, Contains, StartsWith, ToolCalled, StateEquals, and SkillInvoked run without an LLM judge; developers can also extend the Evaluator base class for domain-specific logic. Install the package with pip as strands-agents-evals to use its Python API or the strands-evals command. The CLI can generate experiments, validate experiment JSON, run evaluations, render reports, and diagnose sessions, with documented CI workflow steps. Trace providers can fetch from AWS CloudWatch Logs, Langfuse, and OpenSearch, and session mappers support several tracing formats. The default judge uses Amazon Bedrock with Claude; its example requires authorized credentials. Red-teaming is experimental and may change in a minor release.