Behavioral Modes
Behavioral mode detection is PromptEval's most distinctive capability. It reveals whether your AI produces one consistent type of response or multiple qualitatively different response types — a property no single evaluation can surface.
What a behavioral mode is
A behavioral mode is a cluster of semantically similar responses that your AI produces on a given prompt. If your AI always writes warm, solution-oriented customer service responses, it has one mode. If it sometimes writes warm and detailed responses and sometimes writes terse, transactional ones — and does so in a way that looks random across runs — it has two modes.
Modes emerge from ambiguity. Competing instructions, underspecified directives, or prompts that can be interpreted multiple ways all create conditions where a language model resolves the ambiguity differently on different runs, producing qualitatively distinct response clusters.
Why this matters for debugging
Knowing your AI has two modes changes the engineering problem completely.
One mode, low score
Your AI is consistently mediocre. The fix is to improve overall prompt quality — adjust tone guidance, add few-shot examples, change the system prompt structure.
Two modes, split score
Your AI is excellent in Mode A and poor in Mode B. The fix is to eliminate the ambiguity causing Mode B — find and resolve the competing instructions. Improving "average quality" won't help.
A score of 0.65 could mean either. Without mode detection, you cannot tell which problem you have, so you cannot fix it correctly.
How mode detection works
The pipeline runs seven steps on every multi-run evaluation.
Step 1 — Score each output individually
Every output is scored by the appropriate evaluation channel. You might get scores like: 0.84, 0.29, 0.83, 0.31, 0.85, 0.82, 0.30, 0.84, 0.83, 0.28. Even from the scores alone you can see two clusters — but you can't confirm this from numbers alone.
Step 2 — Embed each output as a semantic vector (SBERT)
Each output text is passed through Sentence-BERT, a model that produces a fixed-size vector where geometric distance in vector space reflects semantic similarity. Two warm, empathetic responses will be close together in this space. A warm response and a terse one will be far apart. SBERT is specifically designed for whole-sentence comparisons, making it well-suited for this task.
Step 3 — Cluster the vectors
K-means clustering is applied to the SBERT vectors. The key challenge is choosing K (how many clusters). PromptEval uses the silhouette coefficient to determine the right K: it measures how tight each cluster is internally and how separated the clusters are from each other. The K that maximizes the silhouette coefficient is selected. If all outputs are semantically similar, K=1 is optimal. If there are two distinct types, K=2 is optimal.
Step 4 — Describe each cluster in plain English
Representative samples from each cluster are sent to the semantic judge with a prompt asking it to describe what the outputs in that cluster have in common. The result is a human-readable description like "warm, empathetic, solution-focused responses" versus "terse, informational, transactional responses."
Step 5 — Compute confidence intervals
A statistically valid confidence interval is computed on the pass rate using the Wilson score interval (for binary pass/fail) or bootstrap resampling (for continuous scores). See Uncertainty Quantification.
Step 6 — Detect distribution shifts
The current output distribution is compared against the stored baseline using a Kolmogorov-Smirnov test. See Regression Detection.
Step 7 — Assemble the diagnostic
All of the above context — mode descriptions, frequencies, confidence interval, distribution shift result — is assembled into a plain-English diagnostic that identifies what changed, likely why, and what to try next.
Minimum runs for reliable mode detection
Clustering requires enough data to distinguish real clusters from noise. With fewer than six or seven runs, results are unreliable. PromptEval defaults to 10 runs as a practical minimum. For robust mode detection, especially when you suspect a third mode might exist, 15–20 runs is better.
Note: The "runs" are runs of your AI, not of PromptEval's evaluation models. PromptEval's judge runs with deterministic settings and produces the same score for the same output every time. The variation across runs comes entirely from your AI.