OpenAI Evals

What OpenAI Evals is

OpenAI Evals is an open-source platform for evaluating model outputs against style and content criteria you define. An evaluation combines a representative test dataset, a data-source schema, and graders that determine whether responses meet requirements. Available graders include string checks, text similarity, score models, label models, and Python code execution. You can configure runs in the OpenAI Dashboard or create and manage them through the Evals API. Runs are asynchronous, support larger data volumes, and allow performance monitoring across versions. Evaluations can target OpenAI models, listed third-party providers, or custom model endpoints. Private evaluations can use your own data without publishing it. The local package installs with pip and requires Python 3.9 or newer. Third-party model use requires an eligible organization, administrator enablement, and acceptance of the usage disclaimer; tool calls are unsupported with external models. Running evals requires an OpenAI API key and incurs model usage costs. The service is scheduled to become read-only on October 31, 2026, and shut down on November 30, 2026.