Guides

Choosing Run Count

More runs produce narrower confidence intervals and more reliable mode detection — but also cost more and take longer. Here's how to pick the right number for your situation.

The fundamental tradeoff

Every additional run narrows your confidence interval and improves the reliability of mode detection. But the relationship is not linear — the marginal benefit diminishes as N grows. Going from 5 to 20 runs is transformative. Going from 100 to 200 runs is modest.

The right question is not "how many runs is enough?" but "how narrow does my CI need to be for the decision I'm making?" a ship/don't-ship decision with a hard threshold needs a tighter CI than an exploratory investigation of a new prompt.

Reference: CI width by run count

Wilson 95% confidence intervals, assuming 80% observed pass rate (8/10, 16/20, etc.).

Runs	CI width	Lower bound	Appropriate for
5	±30%	~36%	Almost nothing — avoid
10	±20%	~49%	Initial exploration; quick CI checks
20	±15%	~60%	Pre-ship validation for most teams
50	±9%	~70%	High-confidence regression detection
100	±6%	~74%	Critical production decisions

Decision guide

Development iteration (exploring a prompt)

Use 10 runs.

You're not making a ship decision — you're checking whether the change moves things in the right direction. A wide CI is acceptable. Focus on the mode count and the pass rate trend.

Pre-ship validation (standard feature)

Use 20–30 runs.

You need enough confidence to make a go/no-go call. At 20 runs and 80% observed pass rate, your lower CI bound (~60%) is still wide. If the gap between your lower bound and threshold is concerning, consider 30 or 50 runs.

Critical system (healthcare, finance, legal)

Use 50–100 runs.

For high-stakes applications where failures have real consequences, require the lower CI bound to clear the threshold, not just the pass rate. This typically requires 50+ runs at most reasonable thresholds.

Major model upgrade (new provider or version)

Use 50–100 runs.

Model behavior can shift in subtle ways. You want the distribution shift test (KS test) to have enough power to detect small but real differences. With fewer than 30 runs per distribution, the test struggles to detect moderate shifts.

Cost reference

The cost of a PromptEval multi-run evaluation is approximately $0.001 per semantic evaluation (one output × one semantic assertion), plus your underlying model costs.

Runs	PromptEval cost (1 semantic assertion)	Your model cost (GPT-4o, ~1k tokens/call)
10	~$0.01	~$0.15
20	~$0.02	~$0.30
50	~$0.05	~$0.75
100	~$0.10	~$1.50

The dominant cost is almost always your underlying model inference, not PromptEval's evaluation. If cost is a constraint, consider using a cheaper model (GPT-4o-mini, Claude Haiku) for evaluation runs, then running fewer production-grade evaluations.