Evaluation Channels
PromptEval routes each assertion to the most appropriate evaluation method. Using a semantic judge for a word-count check wastes time and money. Using a regex for an empathy check produces meaningless results. Routing exists to match tool to task.
Channel overview
| Channel | Assertion type | Cost | Latency |
|---|---|---|---|
| Deterministic | JSON validity, length, string match | Free | <1ms |
| Encoder | Toxicity, PII, language, sentiment | ~$0.0001 | 10–50ms |
| Semantic judge | Empathy, reasoning, task completion | ~$0.001 | 100–500ms |
| UQ layer | Multi-run statistical analysis | ~$0.01–0.02 / 10 runs | 500ms–2s |
Channel 1: Deterministic checks
For assertions that can be answered with code — JSON schema validation, word count, string presence/absence, URL format, language detection — PromptEval runs the check locally inside the SDK. No API call is made. The check is free, takes less than a millisecond, and is perfectly reproducible: the same output always produces the same result.
There is no reason to involve any model in a check like "response is valid JSON." A JSON schema validator is more reliable and infinitely cheaper.
// Assertion type signatures that route here:
{ type: "valid-json" }
{ type: "max-words", value: 400 }
{ type: "contains", value: "Order #12345" }
{ type: "not-contains", value: "competitor_name" }
{ type: "language", value: "en" }
Channel 2: Encoder classification
For well-defined classification tasks — "is this toxic?", "does this contain PII?", "is the sentiment positive?" — PromptEval routes to fine-tuned encoder models: DeBERTa-v3-large or ModernBERT depending on input length.
These are not generic BERT from 2018. DeBERTa-v3 uses a disentangled attention mechanism that processes content and position separately, significantly improving classification accuracy. ModernBERT (released December 2024, trained on 2 trillion tokens) supports sequences up to 8,192 tokens — important for evaluating longer outputs.
For tasks with a fixed label taxonomy, a fine-tuned encoder model reliably outperforms general-purpose LLM judges because it has been specifically trained on that classification problem. Using GPT-4 to check for toxicity is slower, more expensive, and less reliable than using a model trained for exactly that task.
// Assertion types that route here:
{ type: "not-toxic" }
{ type: "no-pii" }
{ type: "sentiment", value: "positive" }
Channel 3: Semantic judge
For assertions that require understanding meaning, context, and nuance — "response demonstrates genuine empathy", "explanation is clear to a non-technical reader", "response matches the specified brand voice" — PromptEval routes to Glider, a fine-tuned version of Microsoft's Phi-3.5-Mini specifically trained for evaluation tasks.
The distinction from a generic small model matters. Glider has been trained on large datasets of scored evaluation judgments and rubrics. Research shows fine-tuned evaluation judges consistently outperform their base models on evaluation tasks, and are on par with much larger general-purpose models on standard semantic evaluation.
Glider runs on PromptEval's self-hosted GPU infrastructure. No data is transmitted to OpenAI, Anthropic, or any third-party server. Inference settings are controlled, making scores fully deterministic — the same output always receives the same score.
// Assertion types that route here:
{ type: "semantic", criterion: "demonstrates genuine empathy", threshold: 0.75 }
{ type: "semantic", criterion: "answer is factually correct given context", threshold: 0.85 }
Honest comparison
For a single evaluation call, Glider is on par with GPT-4 for standard semantic evaluation tasks — not meaningfully better or worse. The reasons to use PromptEval over calling GPT-4 directly are: 15–20× lower cost, deterministic scores, and no data leaving your infrastructure.
Channel 4: The UQ statistical layer
The UQ layer is not a separate routing option — it activates automatically when you provide multiple outputs (via runs: N or by sending pre-collected outputs). It runs on top of whichever channel handles the individual evaluations.
It is what differentiates PromptEval from any other evaluation tool. See Behavioral Modes and Uncertainty Quantification for a complete explanation.