Guides

Interpreting Results

How to read a PromptEval result and decide whether to ship, investigate, or reject.

Step 1 — Check assertion results

Start with the assertions array. Deterministic assertions that fail are hard failures — something structurally wrong with the outputs. Semantic assertion failures are soft signals that require UQ context to interpret.

All assertions passed

Move to the UQ block to understand the distribution. A passing mean score with wide modes or a wide CI might still warrant attention.

Semantic assertion failed

Check the UQ block. If modeCount > 1, the failing mean may be driven by a second behavioral mode — the problem is instability, not overall poor quality. The diagnostic will say so.

Deterministic assertion failed

The problem is structural — invalid JSON, too many words, presence of a banned string. Fix this before interpreting semantic scores.

Step 2 — Check the mode count

uq.modeCount is the first signal from the UQ block. It tells you whether your AI has consistent behavior or multiple distinct behaviors.

modeCount	What it means
1	Consistent behavior. Quality variance is random, not structural. Focus on the CI and the mean score.
2	Two distinct behaviors. Check the mode descriptions and frequencies to understand which mode is dominant. If Mode B occurs >20% of the time, investigate competing instructions.
3+	High behavioral instability. The prompt is ambiguous enough that the model resolves it in three or more qualitatively different ways. Significant prompt engineering work needed.

Step 3 — Read the confidence interval

uq.ci95 gives you the range within which the true pass rate almost certainly falls. Gate on the lower bound, not the pass rate.

ci95[0] > thresholdShip

The lower bound of your CI exceeds the threshold. Even in a pessimistic scenario, you pass. You have statistical evidence the AI reliably meets the bar.

passRate > threshold but ci95[0] < thresholdCollect more runs

The point estimate crosses the threshold but the CI is too wide to be confident. The true rate could be below threshold. Run 30–50 iterations before deciding.

ci95[1] < thresholdDo not ship

Even the upper bound of your CI is below threshold. The AI is reliably failing the criterion. Investigate the diagnostic and fix the prompt.

Step 4 — Check the baseline comparison

If a baseline exists, uq.baselineComparison tells you whether your change caused a statistically significant shift.

significant: false

No significant shift. Score differences are within normal sampling variance. The change you made did not meaningfully affect behavior.

significant: true, direction: "regression"

The distribution has shifted in a statistically significant negative direction. Read the diagnostic — it will describe what changed and why. This is the case where you should investigate before shipping.

significant: true, direction: "improvement"

The distribution has shifted in a statistically significant positive direction. Your change measurably improved behavior. Promote this baseline as the new reference.

Step 5 — Read the diagnostic

The diagnostic field is a plain-English synthesis of everything above. When the findings are significant, it provides:

→A description of what changed (mode count, pass rate, CI width)
→A comparison to the baseline (what the distribution looked like before)
→A probable cause analysis based on the structural differences between modes
→A concrete suggested fix

When the evaluation is clean — one mode, CI above threshold, no significant shift — the diagnostic is brief: "Evaluation passed. One behavioral mode detected. CI clears threshold at lower bound."