Pythia

What it does

Backends

The repository README says Pythia includes Gemini and Ollama backends and can use another backend whose LLM is called with an .invoke() method.checked 5 Oct 2026

Clinical focus

The project describes its optimization as clinical and sensitive to sensitivity and specificity.checked 11 Oct 2026

Configurable thresholds

The README says sensitivity and specificity thresholds are configurable and default to 0.75, while the priority metric defaults to specificity.checked 5 Oct 2026

Data requirements

The documented workflow requires development and validation datasets, an output directory, an SOP, and an initial prompt.checked 11 Oct 2026

Dataset format

The README recommends CSV datasets, with each CSV representing a person and rows representing visits or notes, using “visit” and “Ground Truth” columns.checked 5 Oct 2026

Error analysis

A specialist LLM analyzes failures in aggregate, identifies root causes, and proposes improvement hypotheses.checked 11 Oct 2026

Evaluation

Pythia evaluates prompts against a held-out dataset using configurable metrics and tracks results across iterations.checked 11 Oct 2026

Framework

The site says Pythia uses LangGraph stateful agentic workflows.checked 5 Oct 2026

History and export

Pythia logs prompt versions, scores, and controller decisions, and the site says users can compare runs, export results, and roll back to earlier checkpoints.checked 5 Oct 2026

Hosting and privacy

The project describes Pythia as locally hosted and privacy-preserving, with optimization running on open-source models within the institution.checked 11 Oct 2026

Intended users

The project presents Pythia for clinical prompt optimization and also gives a general example involving home-buying inquiries.checked 11 Oct 2026

Iteration control

A controller can continue, backtrack to a stronger prior prompt, or halt when the prompt has converged.checked 11 Oct 2026

Local hosting

The maker describes Pythia as locally hosted and privacy-preserving, with optimization running on open-source models within the institution.checked 5 Oct 2026

Model backends

The repository says Pythia includes Gemini and Ollama backends and can use another LLM backend that provides an invoke() method.checked 11 Oct 2026

Model scale

The site says optimization can run on models ranging from one billion to one trillion parameters.checked 5 Oct 2026

Prompt improvements

An improver agent generates and ranks multiple candidate prompt rewrites before evaluation.checked 11 Oct 2026

Purpose

Pythia is an automated prompt optimization engine that iteratively evaluates prompts, analyzes failures, and synthesizes improvements.checked 11 Oct 2026

Run history

Pythia logs prompt versions, scores, and controller decisions, and supports comparing runs, exporting results, and rolling back to checkpoints.checked 11 Oct 2026

Support

The project links to its GitHub repository issues for questions and issue reporting.checked 11 Oct 2026

Usage limit

The repository warns that Pythia is API-call heavy and says fewer iterations or a smaller dataset can reduce token costs and computation.checked 11 Oct 2026

Workflow

Its workflow evaluates prompts on a held-out dataset, analyzes failure patterns, proposes prompt rewrites, and controls whether to continue, backtrack, or stop.checked 5 Oct 2026