PromptPex

What PromptPex is

PromptPex is a free, open-source tool for developers who keep AI prompts in their codebase and want to test model behavior. It treats prompts as functions, extracts output rules written in natural language, and generates unit tests for those requirements, such as expected JSON formatting. PromptPex evaluates prompt performance across models and uses an LLM to judge whether outputs satisfy stated conditions. Test generation can be configured with scenarios, test expansions, custom instructions, and effort levels. The CLI accepts Markdown or text templates, Prompty files, and GitHub Models prompt YAML files. Setup materials list OpenAI, Azure OpenAI, GitHub Models, and Ollama providers. The project runs as an npm command-line tool or in Docker, and documentation describes a local web interface. Integrations include OpenAI Evals, Azure OpenAI Stored Completions, GitHub Models Evals, and GitHub Actions. Microsoft is identified as the maker, and the project is intended for research.