PromptEval
API Reference

evaluate()

The primary method for running evaluations. Accepts a prompt, a model function, optional run count, and one or more assertions. Returns detailed results including UQ analysis when multiple runs are requested.

TypeScript signature

types.ts
1async evaluate(options: EvaluateOptions): Promise<EvaluateResult>
2
3interface EvaluateOptions {
4 // The input prompt sent to your model on every run
5 prompt: string
6
7 // Your AI function — receives the prompt, returns a string
8 model: (prompt: string) => Promise<string>
9
10 // Number of times to run the model. Default: 1
11 // When > 1, UQ analysis is included in the result
12 runs?: number
13
14 // Assertions to check against each output
15 assertions: Assertion[]
16
17 // Optional: tag this run for baseline comparison
18 version?: string
19
20 // Optional: compare against a specific tagged baseline instead of the most recent
21 compareAgainst?: string
22
23 // Optional: pre-collected outputs to evaluate instead of running the model
24 // When set, 'model' and 'runs' are ignored
25 outputs?: string[]
26}

Basic usage (single run)

eval.ts
1import { PromptEval } from "@prompteval/sdk"
2
3const client = new PromptEval()
4
5const result = await client.evaluate({
6 prompt: "Summarize the following article in two sentences: {{article}}",
7 model: async (prompt) => gpt4.complete(prompt),
8 assertions: [
9 { type: "max-words", value: 60 },
10 { type: "semantic", criterion: "summary is factually accurate", threshold: 0.85 },
11 ],
12})
13
14console.log(result.assertions[1].score) // e.g. 0.89
15console.log(result.assertions[1].passed) // true

Multi-run evaluation (with UQ)

eval-uq.ts
1const result = await client.evaluate({
2 prompt: "A customer says: 'My order is late. I'm frustrated.'",
3 model: async (prompt) => supportBot.respond(prompt),
4 runs: 20,
5 version: "v1.4.0",
6 assertions: [
7 { type: "not-toxic" },
8 { type: "max-words", value: 400 },
9 { type: "semantic", criterion: "demonstrates genuine empathy", threshold: 0.75 },
10 ],
11})
12
13// UQ fields are present when runs > 1
14console.log(result.uq.modeCount) // e.g. 2
15console.log(result.uq.passRate) // e.g. 0.80
16console.log(result.uq.ci95) // e.g. [0.59, 0.93]
17console.log(result.diagnostic) // plain-English explanation

Evaluating pre-collected outputs

If you've already generated outputs (e.g., from a production system or a batch job), you can pass them directly without running the model again:

eval-outputs.ts
1const result = await client.evaluate({
2 prompt: "Original prompt used to generate these outputs",
3 outputs: [
4 "I completely understand how frustrating this must be...",
5 "Your order is delayed. I'll send an update in two hours.",
6 "I'm so sorry to hear that. Let me look into this right away...",
7 // ... up to 100 pre-collected outputs
8 ],
9 assertions: [
10 { type: "semantic", criterion: "demonstrates genuine empathy", threshold: 0.75 },
11 ],
12})

Python

eval.py
1from prompteval import PromptEval
2import asyncio
3
4client = PromptEval()
5
6async def my_model(prompt: str) -> str:
7 return await gpt4.complete(prompt)
8
9result = asyncio.run(client.evaluate(
10 prompt="A customer says: 'My order is late...'",
11 model=my_model,
12 runs=10,
13 assertions=[
14 {"type": "not-toxic"},
15 {"type": "semantic", "criterion": "demonstrates empathy", "threshold": 0.75},
16 ],
17))
18
19print(result.uq.pass_rate)
20print(result.uq.ci95)

Return value

See Results Schema for the full documented response structure.

Error handling

error-handling.ts
1import { PromptEval, RateLimitError, InvalidAssertionError } from "@prompteval/sdk"
2
3try {
4 const result = await client.evaluate({ ... })
5} catch (err) {
6 if (err instanceof RateLimitError) {
7 // Retry after err.retryAfter seconds
8 }
9 if (err instanceof InvalidAssertionError) {
10 // Assertion configuration is malformed
11 console.error(err.field, err.message)
12 }
13}