Traditional software tests are deterministic. The same input always produces the same output. A failing assertion means something broke. A passing test suite means nothing broke. This works perfectly — for traditional software.
AI applications are not deterministic. Published research running a prompt 1,000 times at temperature zero produced 80 unique completions — the most common appeared only 78 times. This means: when you change your system prompt and your tests pass, you know your AI produced an acceptable output that one time. You do not know whether the change shifted your AI's behavioral distribution in ways that will manifest as problems on 10%, 30%, or 40% of future requests.
This is a silent regression. No exception. No error. No alert. Your AI just starts behaving differently in production, affecting real users, until someone notices.
Added "be concise" to system prompt → empathy instructions now lose to brevity 40% of the time
Customer support bot stops recommending doctor contact in 30% of medical queries. No test catches it.
Upgraded from GPT-4 to GPT-4o → tone shifted slightly more casual on formal queries
Luxury brand chatbot violating brand standards on 25% of responses. Passes all existing tests.
Reordered few-shot examples → model infers different implicit behavioral standard
Factual accuracy drops on edge case inputs. Traditional tests never touched those cases.
None of these trigger an exception. None fail a traditional test. All three are caught by PromptEval.
The primary score. Built on UQLM's non-contradiction scorer — an NLI model that compares every pair of responses. No access to model internals required.
Is the inconsistency random, or does your AI have distinct behavioral patterns? Clusters embeddings to find modes — the signature of competing instructions.
The Wilson confidence interval tells you how much to trust the result. The Kolmogorov-Smirnov test identifies statistically significant behavioral shifts.
Every evaluation returns a structured diagnostic explaining what was found — not just a score. When two behavioral modes are detected, you see a sample response from each mode, the frequency of each, and the interpretation of what the pattern means.
{ "result": { "score": 0.51, "passed": false }, "layer_2_behavioral": { "modes_detected": 2, "modes": [ { "label": "warm_empathetic", "frequency": 0.60, "sample": "I completely understand how frustrated you..." }, { "label": "terse_transactional", "frequency": 0.40, "sample": "Your order is delayed. Update in 2 hours." } ] }, "diagnostic": { "summary": "FAIL — Behavioral regression detected...", "detail": "Two distinct behavioral modes detected..." } }
PromptEval is designed to be one layer in a robust AI testing pipeline. It solves for Behavioral Regression, but it is not a replacement for correctness or safety testing.
Measures behavioral consistency. Detects if your model's response distribution has shifted or developed new behavioral modes.
Measures if the output matches a reference ground truth or adheres to specific factual requirements (e.g., DeepEval Factuality).
Measures nuance, brand voice, and subtle safety edge cases that require subjective human judgment.
"Consistency is not correctness. A model that consistently produces the wrong answer is 100% consistent and 0% correct. PromptEval detects when your AI's behavior shifts; you decide if the shift is an improvement."
# PromptEval executes all inference calls in parallel
async def run_consistency_check(prompt, n=5):
# 5 calls run at the same time
responses = await asyncio.gather(
*[llm.generate(prompt) for _ in range(n)]
)
# Results sent to GPU-accelerated NLI scorer
result = await prompteval.score(responses)
return resultTotal latency is roughly the same time as a single LLM call plus ~100ms for batched scoring.
Traditional evals check if a single output is 'good'. PromptEval checks if your system prompt change shifted the entire behavioral distribution.
When moving from GPT-4 to GPT-4o, benchmarks say intelligence stays the same. PromptEval tells you if the nuances of your product's voice actually shifted.
For medical, legal, or financial AI, consistency isn't just a quality metric — it's a requirement. Ensure your AI behaves exactly the same way, every time.
Change system prompt to 'be concise'
Run traditional tests (All Pass)
Deploy to Production
AI behavior shifts; empathy instructions lost
14 days of brand damage until manually detected
Change system prompt to 'be concise'
Run PromptEval CI Gate
Consistency drop detected (v1.0 vs v1.1)
PR Blocked. Behavioral regression averted.
PromptEval doesn't just measure consistency against an abstract threshold. Every test run is compared against a specific version_tag or historical run of your choice.
When you run an eval on a feature branch, PromptEval automatically fetches the distribution of the main branch for comparison. This allows for the Kolmogorov-Smirnov (KS) test to detect if the new version is statistically different from the baseline.
One import into your existing DeepEval or pytest suite. No separate infrastructure to manage or wait for.
Proprietary batched NLI scoring engine runs evaluations significantly faster than sequential LLM calls.
Measure consistency on any model including GPT-4o, Claude 3.5, and Gemini. No logprob or model internal access required.
We generate multiple pairs of responses for every test case to ensure the consistency score is mathematically significant.
Model data never touches our servers. The evaluation runs in your infrastructure, using your existing LLM provider keys.
Every run is stored and indexed by version_tag. Fetch historical distributions for CI comparison in real-time.
Non-contradiction batched scoring engine
Automated behavioral mode detection
Wilson Confidence Intervals
Distribution shift detection
Zero-latency penalty architecture
Visualize regression over time
PromptEval automatically tags every run and allows you to visualize behavioral shifts between versions in real-time.
Export results as JUnit XML or JSON for native integration with GitHub Actions, GitLab CI, and CircleCI.
Our scoring logic isn't a proprietary black box. The NLI methodology and KS test parameters are fully documented and auditable.
The open-source core allows AI researchers to contribute new behavioral detection modes and statistical tests to the community.
The PromptEval SaaS manages the GPU infrastructure and database layer required for high-throughput production evaluation.