PromptEval
Getting Started

Quickstart

Get your first multi-run evaluation running in under five minutes.

1. Install the SDK

terminal
1npm install @prompteval/sdk

Supports Node.js 18+. For Python support, see the Installation page.

2. Set your API key

.env
1PROMPTEVAL_API_KEY=pe_live_...

Get your key from the Dashboard → API Keys. Keys are scoped per project.

3. Write your first evaluation

The example below evaluates a customer-support AI across 10 runs and checks three properties: toxicity, length, and empathy.

eval.ts
1import { PromptEval } from "@prompteval/sdk"
2import { myAI } from "./my-ai" // your existing AI function
3
4const client = new PromptEval({ apiKey: process.env.PROMPTEVAL_API_KEY })
5
6const result = await client.evaluate({
7 // The prompt fed to your AI on every run
8 prompt: "A customer says: 'My order has been stuck for a week. I'm really frustrated.'",
9
10 // A function that takes the prompt and returns a string response
11 model: async (prompt) => myAI.complete(prompt),
12
13 // Number of times to run the model. Default: 10
14 runs: 10,
15
16 assertions: [
17 // Free — checked locally without any API call
18 { type: "not-toxic" },
19 { type: "max-words", value: 400 },
20
21 // Semantic — routed to the fine-tuned judge model
22 {
23 type: "semantic",
24 criterion: "response demonstrates genuine empathy toward the customer",
25 threshold: 0.75,
26 },
27 ],
28})
29
30console.log(result)

4. Read the output

Here's what you'll see in the terminal. Pay attention to the uq block — that's the statistical analysis.

output (annotated)
1{
2 "prompt": "A customer says: 'My order...'",
3 "runs": 10,
4
5 "assertions": [
6 { "name": "not-toxic", "passed": true, "score": 0.02, "channel": "encoder" },
7 { "name": "max-words", "passed": true, "score": null, "channel": "deterministic" },
8 { "name": "empathy", "passed": false, "score": 0.61, "channel": "semantic" }
9 ],
10
11 "uq": {
12 // How many distinct behavioral modes the model has on this prompt
13 "modeCount": 2,
14
15 "modes": [
16 {
17 "id": "A",
18 "frequency": 0.60,
19 "avgScore": 0.84,
20 "description": "Warm, empathetic, solution-focused responses that acknowledge the customer's frustration"
21 },
22 {
23 "id": "B",
24 "frequency": 0.40,
25 "avgScore": 0.29,
26 "description": "Terse, informational responses that address the issue without emotional acknowledgment"
27 }
28 ],
29
30 // Of the 10 runs, 6 passed the empathy threshold of 0.75
31 "passRate": 0.60,
32
33 // Wilson 95% confidence interval on the pass rate
34 // — you cannot be sure the true rate is above 50% yet
35 "ci95": [0.30, 0.84],
36
37 // How this compares to the last stored baseline
38 "baselineComparison": {
39 "previousPassRate": 0.90,
40 "shift": -0.30,
41 "pValue": 0.003,
42 "significant": true
43 }
44 },
45
46 "diagnostic": "A behavioral regression was introduced. Previously your AI had one behavioral mode (warm, empathetic). It now has two distinct modes, with Mode B occurring 40% of the time. The most likely cause is the 'Be concise' instruction added in the latest system prompt edit, which competes with the empathy instruction. Suggested fix: reframe conciseness as subordinate to empathy — e.g. 'Be efficient with words while always maintaining warm, empathetic tone.'"
47}

What this tells you

modeCount: 2

Your AI is not producing one consistent type of response — it's switching between two qualitatively different behaviors. This is the most actionable signal in the output.

ci95: [0.30, 0.84]

The confidence interval is wide. You've observed 6/10 passes, but based on that sample the true pass rate could plausibly be anywhere from 30% to 84%. Run 20–50 iterations to narrow this down before making a ship decision.

significant: true, pValue: 0.003

The distribution has shifted significantly compared to the baseline. This wasn't random noise — something you changed affected behavior.

Next steps

How behavioral mode detection works →Understanding confidence intervals →Guide: interpreting the full output →Running evaluations in CI →