Introduction
PromptEval is an evaluation API for AI-powered applications. It analyzes distributions of outputs, not individual responses — giving you statistically valid insight into how your AI behaves across many runs.
The problem it solves
Language models are stochastic. Given the same prompt, they produce different outputs on different runs — even at temperature zero. This means a single evaluation is a single sample from a distribution. It tells you what happened once. It does not tell you what your AI does reliably.
Consider this scenario: you run one evaluation, get a score of 0.81, clear the threshold of 0.75, and ship. But the underlying distribution might be 40% above threshold and 60% below. Your test passed by luck.
Worse: your AI might have two distinct behavioral modes — sometimes warm and empathetic, sometimes terse and transactional — and a single average score hides both. No other evaluation tool surfaces this.
What PromptEval does
When you run N evaluations of the same prompt, PromptEval does not just average the scores. It runs a full statistical analysis:
Behavioral mode detection
Embeds all outputs as semantic vectors, clusters them, and reports whether your AI has one consistent behavior or multiple distinct modes.
Confidence intervals
Computes Wilson score or bootstrap intervals on your pass rate so you know how much to trust the number.
Distribution shift detection
Compares the current run against a stored baseline using a KS test. Flags statistically significant regressions.
Diagnostic explanations
Combines all of the above into a plain-English diagnosis — what changed, why, and what to try next.
Evaluation channels
PromptEval routes each assertion to the most appropriate evaluation method so you pay only for what you need.
| Channel | Used for | Cost |
|---|---|---|
| Deterministic | JSON validity, word count, string presence | Free |
| Encoder | Toxicity, PII, sentiment classification | ~$0.0001 |
| Semantic judge | Empathy, reasoning quality, task completion | ~$0.001 |
| UQ layer | Multi-run statistical analysis (adds on top) | ~$0.01–0.02 / 10 runs |
When to use PromptEval
- →You're evaluating prompts or models before shipping to production
- →You need to detect regressions when changing system prompts or model versions
- →You want to know whether your AI's quality is consistent or bimodal
- →You need evaluations that don't send data to third-party servers
- →You're running evaluations in CI and need deterministic, reproducible scores
A quick look
Here's what a multi-run evaluation call looks like: