PromptMetrics
PromptMetrics is a free, open-source, self-hosted registry for managing prompts and observing LLM activity.
What PromptMetrics is
PromptMetrics is a free, open-source, self-hosted registry for managing prompts and observing LLM activity. Prompts can be stored, versioned, retrieved, and rolled back through a REST API or CLI, with changes recorded in Git. It logs model, token, latency, cost, and custom-tag metadata, exporting structured data to stdout JSON or OpenTelemetry. Agent traces, spans, workflow runs, and environment-labeled prompt versions are supported. Evaluation suites run against datasets with budget tracking, and A/B tests compare prompt versions and can promote a winner. A compliance engine scans prompts for PII, API keys, URLs, and IP addresses, assigning risk scores. The playground proxies calls for OpenAI, Anthropic, Cohere, Ollama, and Azure OpenAI. Storage drivers include filesystem, GitHub, and S3 for prompt content, with SQLite or PostgreSQL for metadata. Interfaces include an optional Next.js dashboard, CLI, REST API, and Node.js and Python SDKs. Installation is documented through npm, Docker Compose, or source.
Good
- Free and MIT licensed
- Git-backed prompt rollback
- LLM metadata and OpenTelemetry export
- Dataset evaluations and A/B testing
- Node.js and Python SDKs
Watch out
- Self-hosted deployment required
- Default limit is 300 requests per 60 seconds
- Dashboard is optional
- Security guidance requires operational setup
Who it suits
It suits developers and teams that need self-hosted prompt versioning, LLM observability, evaluations, and compliance checks. It fits environments using multiple model providers and Git-backed workflows.
Verdict
PromptMetrics brings prompt management, telemetry, testing, and compliance into a self-hosted project. Its storage choices and provider proxy support make it adaptable, provided teams handle deployment and security configuration.
The facts
Each one read from PromptMetrics's own pages, with the date it was checked.
What it does
- Agent telemetry
- It tracks agent traces and spans, workflow runs, and prompt versions tagged with environment labels.checked 8 Oct 2001
- Compliance scanning
- Its compliance engine scans prompts for PII, API keys, URLs, and IP addresses and assigns risk scores.checked 8 Oct 2001
- Dashboard
- An optional Next.js dashboard provides charts and views for prompts, logs, traces, evaluations, A/B tests, compliance, and the playground.checked 8 Oct 2001
- Dashboard key handling
- The dashboard authenticates through a BFF proxy so API keys are not stored in browser localStorage.checked 8 Oct 2001
- Deployment
- The README provides installation options using npm, Docker Compose, or source, and describes the product as self-hosted.checked 8 Oct 2001
- Evaluations and A/B tests
- Evaluation suites can be run against datasets with budget tracking, and A/B tests compare prompt versions and can promote a winner.checked 8 Oct 2001
- Evaluations and testing
- It supports evaluation suites, dataset-backed evaluation runs, and statistical A/B comparisons with winner promotion.checked 8 Oct 2001
- Installation
- The README documents installation through npm, Docker Compose, or building from source.checked 8 Oct 2001
- Interfaces and SDKs
- The project provides a REST API, CLI, optional Next.js dashboard, and Node.js and Python SDKs.checked 8 Oct 2001
- License
- The repository identifies the project as MIT licensed.checked 8 Oct 2001
- LLM providers
- The playground can proxy chat and completion calls for OpenAI, Anthropic, Cohere, Ollama, and Azure OpenAI.checked 8 Oct 2001
- Logging
- Structured logs can capture model, token, latency, cost, and custom-tag metadata and export to stdout JSON or OpenTelemetry.checked 8 Oct 2001
- Observability
- It logs LLM request metadata such as model, tokens, latency, cost, and custom tags, and can export data through OpenTelemetry.checked 8 Oct 2001
- Prompt versioning
- Prompts can be stored, versioned, retrieved, and rolled back through a REST API or CLI, with changes recorded in Git.checked 8 Oct 2001
- Purpose
- PromptMetrics is a lightweight, self-hosted prompt registry with Git-backed versioning, metadata logging, and evaluations for LLM observability.checked 8 Oct 2001
- Rate limit
- The README lists a default limit of 300 requests per 60-second window per API key, configurable through environment variables.checked 8 Oct 2001
- SDKs
- The project lists Node.js and Python SDKs for programmatic access.checked 8 Oct 2001
- Security controls
- API keys are HMAC-SHA256 hashed, support scoped permissions and optional expiration, and the dashboard uses a proxy so keys are not stored in browser localStorage.checked 8 Oct 2001
- Security guidance
- The security policy recommends HTTPS, a reverse proxy, a long random API key salt, and regular database backups.checked 8 Oct 2001
- Storage
- Prompt content is stored in Git, while metadata can use SQLite or PostgreSQL; filesystem, GitHub, and S3 storage drivers are listed.checked 8 Oct 2001
- Storage options
- Prompt content can use filesystem, GitHub, or S3 storage, while metadata can use SQLite or PostgreSQL.checked 8 Oct 2001
- What it does
- PromptMetrics is a lightweight, self-hosted prompt registry for versioning prompts and tracking LLM activity.checked 8 Oct 2001
The company
- Headquarters
- Berlin, Germanychecked 28 Sep 2001
- Maker and company details
- The repository says it is maintained by the PromptMetrics team; it does not state a headquarters or founding date.checked 8 Oct 2001
Alternatives to PromptMetrics
- 1
Vellum
Free planOpen sourceAndroidiOSWebAPI8.7From $30 a month - 2
Langfuse
Free planOpen sourceWebAPILinuxSelf-hosted8.1From $29 a month - 3
PromptHelm
Free planAndroidiOSWebAPI8.0From $39 a month - 4
Agenta
Free planOpen sourceWebAPISelf-hosted7.9From $29 a month - 5
Promptfoo
Free planOpen sourceWebAPILinuxmacOS7.8Free plan - 6
Braintrust
Free planWebAPISelf-hosted7.6From $249 a month