Uncertainty Quantification
A score without a confidence interval is an incomplete measurement. PromptEval computes statistically valid uncertainty bounds on every multi-run evaluation so you know not just what the score is, but how much to trust it.
Why a single score isn't enough
You run 10 evaluations and 6 pass the threshold. Your point estimate is 60%. But how confident are you that the true pass rate is near 60%? Based on just 10 samples, it could plausibly be anywhere from 30% to 84%. You might be above or below your threshold goal, and your 10 samples don't reliably tell you which.
Confidence intervals are not optional rigor — they are the minimum information needed to make a ship/don't-ship decision responsibly.
Wilson score interval (pass rate)
For binary pass/fail rates — "what fraction of runs passed the threshold?" — PromptEval uses the Wilson score interval. This is the statistically recommended method for computing confidence intervals on proportions from small samples.
The naive alternative (the normal approximation) behaves poorly when the proportion is near 0 or 1 or the sample size is small — both common in evaluation scenarios. The Wilson interval handles both correctly.
| Observed | Point estimate | Wilson 95% CI |
|---|---|---|
| 6 / 10 | 60% | [30%, 84%] — high uncertainty |
| 9 / 10 | 90% | [59%, 98%] — probably reliable |
| 18 / 20 | 90% | [70%, 97%] — narrowing |
| 45 / 50 | 90% | [79%, 96%] — solid |
| 90 / 100 | 90% | [83%, 95%] — trustworthy |
The CI narrows as you collect more runs. Whether 10 runs is enough depends on how narrow your CI needs to be for the decision you're making. See Choosing Run Count.
Bootstrap CI (continuous scores)
For continuous score distributions — the actual numeric empathy scores across all runs, not just pass/fail — PromptEval uses bootstrap resampling:
- Draw 10,000 random samples of size N from the observed scores (with replacement).
- Compute the mean of each resample.
- Take the 2.5th and 97.5th percentiles as the 95% confidence bounds.
Bootstrap is distribution-agnostic — it does not assume the underlying distribution is normal — making it appropriate for the skewed, multimodal distributions that AI evaluation scores often produce.
How to read the confidence interval
The 95% CI means: if you repeated this entire experiment many times, 95% of the computed intervals would contain the true pass rate. In practical terms: the true pass rate is almost certainly inside the interval you're looking at.
Wide CI → collect more runs
ci95: [0.30, 0.84] — You cannot make a confident ship decision. The true rate could be 30% (bad) or 84% (acceptable). Run 30–50 iterations.
Both bounds above threshold → confident pass
ci95: [0.78, 0.97] with threshold 0.75 — Even in the pessimistic scenario, you're above threshold. Ship.
Both bounds below threshold → confident fail
ci95: [0.21, 0.55] with threshold 0.75 — The AI is reliably below threshold. Fix the prompt, don't ship.