Core Concepts

Regression Detection

PromptEval stores the output distribution from every evaluation run and uses statistical testing to detect when a change has caused your AI's behavior to shift significantly.

How it works

Every time you call evaluate(), PromptEval records the full output distribution — the scores across all N runs, the mode structure, and the cluster centroids — against your project, assertion name, and optionally a version tag you provide.

On subsequent evaluations, PromptEval compares the new distribution against the stored baseline using a two-sample Kolmogorov-Smirnov (KS) test. The KS test asks a specific question: are these two sets of measurements consistent with being samples from the same underlying distribution, or have they diverged?

If the p-value falls below 0.05, the distributions are statistically different. PromptEval reports this as a significant shift, includes the shift direction and magnitude, and adds it to the diagnostic explanation.

The Kolmogorov-Smirnov test

The KS test is a non-parametric test that compares two cumulative distribution functions. It does not assume that the scores are normally distributed — important because AI evaluation score distributions are often skewed or multimodal.

The test computes the maximum distance between the two CDFs. If this distance exceeds a critical value (determined by the sample sizes and significance level), the distributions are considered statistically different.

Interpreting the p-value

p > 0.05 — No significant shift detected. Score change is consistent with normal sampling variance.
p < 0.05 — Significant shift detected. The distribution changed in a non-random way.
p < 0.01 — Strong evidence of shift. If the score went down, something you changed caused it.

Why "the score went down" isn't enough

If you evaluate before and after a change, you might observe the average score dropping from 0.78 to 0.72. Is this a real regression or sampling noise? Without statistical testing you cannot tell.

False alarm (no testing)

Score drops from 0.78 to 0.72. You roll back the change, spend two hours debugging. The KS test would have told you p=0.43 — normal variance, not a real shift.

Missed regression (no testing)

Score drops from 0.78 to 0.74. Stays above threshold. You ship. The KS test would have shown p=0.002 — a real shift. The distribution shifted but didn't cross threshold on this run.

Baseline management

By default, PromptEval uses the most recent stored evaluation as the baseline for comparison. You can also tag evaluations with a version label and compare against a specific tagged baseline:

regression-check.ts

1// Tag this run as "v2.1.0"
2const result = await client.evaluate({
3  prompt: "...",
4  model: async (p) => myAI.complete(p),
5  runs: 20,
6  version: "v2.1.0",
7  assertions: [
8    { type: "semantic", criterion: "demonstrates empathy", threshold: 0.75 },
9  ],
10})
11 
12// Compare against a specific previous tag
13const result = await client.evaluate({
14  prompt: "...",
15  model: async (p) => myAI.complete(p),
16  runs: 20,
17  compareAgainst: "v2.0.0",  // compare against v2.0.0 baseline
18  assertions: [...],
19})

In the API response

result.uq.baselineComparison

1{
2  "baselineComparison": {
3    "baselineVersion": "v2.0.0",
4    "baselinePassRate": 0.90,
5    "currentPassRate": 0.60,
6    "shift": -0.30,
7    "pValue": 0.003,
8    "significant": true,
9    "direction": "regression"
10  }
11}

When significant: true and direction: "regression", the diagnostic will explain the probable cause based on what changed in the evaluation context.