Guides

CI / CD Integration

Run PromptEval evaluations on every pull request to catch behavioral regressions before they reach production. Because PromptEval's judge is deterministic, the same output always produces the same score — evaluations are reproducible in CI.

Recommended CI workflow

Run evaluations before and after each change (use the version tag to label the commit).

Fail the build if a statistically significant regression is detected (p < 0.05, significant: true, direction: 'regression').

Gate on the lower bound of the CI, not the pass rate. If ci95[0] < your threshold, the confidence interval doesn't clear — require more runs or reject.

Store evaluation results as CI artifacts for audit and trend visibility.

GitHub Actions

.github/workflows/eval.yml

1name: Prompt Evaluation
2 
3on:
4  pull_request:
5    paths:
6      - "prompts/**"
7      - "src/ai/**"
8 
9jobs:
10  evaluate:
11    runs-on: ubuntu-latest
12    steps:
13      - uses: actions/checkout@v4
14 
15      - uses: actions/setup-node@v4
16        with:
17          node-version: 20
18 
19      - run: npm ci
20 
21      - name: Run PromptEval
22        env:
23          PROMPTEVAL_API_KEY: ${{ secrets.PROMPTEVAL_API_KEY }}
24          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
25        run: npx tsx evals/run.ts
26 
27      - name: Upload results
28        if: always()
29        uses: actions/upload-artifact@v4
30        with:
31          name: eval-results
32          path: eval-results.json

The evaluation script

evals/run.ts

1import { PromptEval } from "@prompteval/sdk"
2import { supportBot } from "../src/ai/support-bot"
3import { writeFileSync } from "fs"
4 
5const client = new PromptEval()
6 
7const result = await client.evaluate({
8  prompt: "A customer says: 'My order has been stuck for a week. I'm frustrated.'",
9  model: async (prompt) => supportBot.respond(prompt),
10  runs: 20,
11  version: process.env.GITHUB_SHA, // tag with commit SHA for traceability
12  assertions: [
13    { type: "not-toxic" },
14    { type: "max-words", value: 300 },
15    {
16      type: "semantic",
17      criterion: "demonstrates genuine empathy toward the customer",
18      threshold: 0.75,
19    },
20  ],
21})
22 
23// Save results as CI artifact
24writeFileSync("eval-results.json", JSON.stringify(result, null, 2))
25 
26// Fail the job if:
27// 1. A statistically significant regression was detected, OR
28// 2. The lower CI bound is below the threshold (not enough confidence)
29const empathyAssertion = result.assertions.find((a) => a.name === "empathy")
30const uq = result.uq!
31 
32const significantRegression =
33  uq.baselineComparison?.significant &&
34  uq.baselineComparison.direction === "regression"
35 
36const ciTooLow = uq.ci95[0] < 0.75  // lower bound < threshold
37 
38if (significantRegression) {
39  console.error("❌ Behavioral regression detected:", result.diagnostic)
40  process.exit(1)
41}
42 
43if (ciTooLow) {
44  console.error(
45    `❌ Confidence interval lower bound (${uq.ci95[0].toFixed(2)}) is below threshold (0.75). ` +
46    `Pass rate may not be reliable. Run more iterations or investigate distribution.`
47  )
48  process.exit(1)
49}
50 
51console.log(`✅ Evaluation passed. Pass rate: ${(uq.passRate * 100).toFixed(0)}%, CI: [${uq.ci95.map(v => v.toFixed(2)).join(', ')}]`)
52process.exit(0)

GitLab CI

.gitlab-ci.yml

1evaluate:
2  image: node:20
3  stage: test
4  only:
5    changes:
6      - prompts/**/*
7      - src/ai/**/*
8  variables:
9    PROMPTEVAL_API_KEY: $PROMPTEVAL_API_KEY
10    OPENAI_API_KEY: $OPENAI_API_KEY
11  script:
12    - npm ci
13    - npx tsx evals/run.ts
14  artifacts:
15    when: always
16    paths:
17      - eval-results.json

Secret management

Store your PROMPTEVAL_API_KEY as a repository secret. Create a separate key per environment (dev/prod) so you can revoke CI access independently. Keys scoped to a project have no access to other projects.