LLM outputs are non-deterministic and rarely have one right answer, so you can't unit-test them the old way. Evals replace vibes with evidence: curated datasets, deterministic and semantic scorers, and an LLM-as-judge that grades free-form text against a rubric - calibrated against humans and wired into CI so quality never silently regresses.
An eval is a repeatable measurement of how well an LLM system does a task: run it over a fixed dataset, score each output with one or more graders, and aggregate into metrics you can track over time. Because outputs are open-ended, scoring often blends deterministic checks (exact match, JSON validity) with semantic and LLM-as-judge grading. The goal is the same as any test suite - catch regressions and prove improvements - but adapted to text that has no single correct string.
A curated set of inputs plus optional references or expected behaviors - golden examples, sampled real traffic, and hand-built edge cases.
Functions that turn an output into a score: deterministic rules, semantic similarity, task-specific metrics, or an LLM judge with a rubric.
Aggregated metrics per slice, compared to a baseline, so you can see pass rate, faithfulness, and where quality moved up or down.
| Piece | What it does | Common choices |
|---|---|---|
| Eval dataset | Fixed inputs (+ references) to run the system against | Golden set, sampled traffic, synthetic edge cases |
| Deterministic scorer | Rule-based pass/fail, cheap and exact | Exact match, regex, JSON.parse, schema valid |
| Semantic scorer | Measures meaning-level closeness to a reference | Embedding cosine, BERTScore, ROUGE |
| Task-specific scorer | Metrics tied to the job (esp. RAG) | Faithfulness, answer-relevance, context recall |
| LLM-as-judge | Grades open-ended text with a rubric or reference | Pairwise, reference-free, reference-based |
You can't ship what you can't measure. LLM behavior shifts with every prompt tweak, model upgrade, or retrieval change - and it changes silently. Evals turn "seems better" into a number, and stop yesterday's fix from becoming tomorrow's regression.
The same prompt yields different text each run. Evals average over a dataset so you measure a distribution, not one lucky (or unlucky) sample.
Many tasks have many acceptable outputs. Rubric-based judging scores quality and correctness properties instead of demanding one exact string.
A prompt edit that helps one case can break ten others. Running evals in CI blocks the regression before it reaches users.
Comparing candidate versions on the same dataset gives a defensible reason to promote one prompt, model, or retriever over another.
Same idea, four ways to picture it, so it clicks whoever you are.
An eval is the graded exam with an answer key and a rubric. You don't judge a single essay by gut feeling; you run every student against the same questions and mark each one against explicit criteria for full, honest credit.
It's a test suite for prose. Deterministic checks are your assertions; the LLM-as-judge is a fuzzy matcher for text that has no one correct string. Wire it into CI and a regression fails the build like any broken test.
Think of a QA reviewer with a scorecard. Instead of spot-checking a few calls by vibe, the reviewer samples a representative set and scores each against the same quality standard, so decisions rest on evidence, not anecdotes.
Like a cooking competition judge with a rubric card: taste, presentation, technique. The rubric keeps every dish scored the same way, so the winner isn't just whoever plated last or served the biggest portion.
Evals run in two settings: offline against a curated dataset (before you ship), and online against sampled production traffic (after you ship). Both feed the same scorers; only the source of inputs differs.
Take a fixed dataset, run the system under test on every row, score each output, and diff the aggregate metrics against a stored baseline. Deterministic, reproducible, gate-able in CI.
Sample live traffic, run reference-free judges and heuristics on real outputs, and track metrics on dashboards. Catches drift and edge cases your dataset never anticipated.
flowchart LR
subgraph Data["🗂️ Eval dataset"]
G["⭐ Golden set"] --> DS[("📚 Cases + refs")]
T["📡 Sampled traffic"] --> DS
E["🧨 Edge cases"] --> DS
end
subgraph Run["🧪 Eval harness"]
DS --> SUT["🤖 System under test"]
SUT --> OUT["📝 Candidate outputs"]
OUT --> DET["📏 Deterministic scorers"]
OUT --> SEM["🧬 Semantic scorers"]
OUT --> JUD["⚖️ LLM-as-judge"]
DET --> AGG["📊 Aggregate + slice"]
SEM --> AGG
JUD --> AGG
end
AGG --> REP["📈 Report vs baseline"]
You are a strict grader. Score the ANSWER on each criterion
from 1-5. Judge only what is written. Do NOT reward length.
Return JSON only.
Criteria:
- faithfulness: every claim is supported by CONTEXT (no invented facts)
- relevance: the answer addresses the QUESTION directly
- completeness: no essential part of the question is ignored
CONTEXT: {{context}}
QUESTION: {{question}}
ANSWER: {{answer}}
Respond exactly as:
{"faithfulness": n, "relevance": n, "completeness": n,
"rationale": "one sentence"}Notice the guardrails: a fixed scale, "do not reward length" to fight verbosity bias, JSON-only output for deterministic parsing, and a forced rationale so scores are auditable. A judge is only trustworthy once its scores are calibrated against human labels.
Three views: an offline harness run, an LLM-as-judge scoring one output against a rubric, and a CI pipeline that blocks a deploy on regression.
sequenceDiagram
autonumber
participant Dev as 👤 Engineer
participant Harn as 🧪 Eval harness
participant DS as 🗂️ Dataset
participant SUT as 🤖 System under test
participant Scr as 📏 Scorers
participant Rep as 📈 Report
Dev->>Harn: run eval(suite, baseline)
Harn->>DS: load cases + references
DS-->>Harn: N test rows
loop for each case
Harn->>SUT: run(input)
SUT-->>Harn: candidate output
Harn->>Scr: score(output, reference)
Scr-->>Harn: metric values
end
Harn->>Rep: aggregate + diff vs baseline
Rep-->>Dev: pass rate, deltas, failures
sequenceDiagram
autonumber
participant Harn as 🧪 Harness
participant Prompt as 🧩 Rubric builder
participant Judge as ⚖️ Judge LLM
participant Cal as 🎚️ Calibration
participant Store as 🗄️ Results
Harn->>Prompt: build(question, answer, reference, rubric)
Prompt-->>Harn: judge prompt
Harn->>Judge: grade(prompt)
Judge-->>Harn: scores + rationale (JSON)
Note over Harn,Judge: swap A/B order to check position bias
Harn->>Cal: compare to human labels
alt agreement below threshold
Cal-->>Harn: recalibrate rubric
else judge trusted
Cal-->>Harn: accept scores
end
Harn->>Store: persist scores + rationale
sequenceDiagram
autonumber
participant Dev as 👤 Developer
participant CI as 🔁 CI pipeline
participant Harn as 🧪 Eval harness
participant Base as 📊 Baseline metrics
participant Deploy as 🚀 Deploy
Dev->>CI: open pull request
CI->>Harn: run eval suite on PR build
Harn-->>CI: candidate metrics
CI->>Base: compare against threshold
alt metrics regressed
Base-->>CI: fail (delta below gate)
CI-->>Dev: block merge + show diff
else metrics hold or improve
Base-->>CI: pass
CI->>Deploy: promote build
Deploy-->>Dev: shipped
end
From "we have no idea if it's good" to a calibrated judge guarding every deploy - the whole journey in order.
Write down the task and the qualities that matter - correctness, faithfulness, tone, safety. Vague goals produce useless evals.
Curate representative inputs with expected answers or behaviors. Start small (30–100 hand-checked cases) but make them count.
Sample production logs for realistic distribution, then hand-craft adversarial and rare cases the golden set misses.
Use deterministic checks where an exact/JSON answer exists; semantic similarity for paraphrase; a judge for open-ended quality.
Define explicit criteria and a scoring scale, forbid length rewards, and demand JSON plus a rationale for auditability.
Have humans label a slice, then measure judge–human agreement. Tune the rubric until they correlate; otherwise the judge lies.
Swap A/B order to counter position bias, cap or normalize length, and avoid self-preference by judging with a different model family.
Execute the full suite to get baseline metrics per slice. This frozen snapshot is what every future change is measured against.
Run the suite on every PR and fail the build when key metrics drop below the baseline threshold - a hard regression gate.
Sample live outputs, run reference-free judges and heuristics, and dashboard the trends to catch drift the offline set can't.
Promote real failures into the dataset, refine rubrics, and re-baseline. The eval suite compounds in value with every cycle.
Most eval failures come from the dataset and the judge, not the model under test. These are the usual culprits.
Ten cherry-picked cases don't cover real traffic. A high pass rate on a toy set proves nothing; the failures you never sampled are exactly the ones that ship to users.
Judges quietly favor the first option (position), the longer answer (verbosity), and outputs from their own model family (self-preference). Unswapped, un-normalized judging bakes those biases straight into your scores.
A judge nobody checked against humans is a random number generator with a rationale. Without measured judge-human agreement, you don't know whether a "4/5" means anything at all.
Optimizing one aggregate number invites Goodhart's law: the system learns to win the metric while real quality slips. One score hides regressions in every slice it averages over.
When eval cases leak into the prompt, few-shot examples, or fine-tuning data, the system is graded on answers it already saw. Scores soar and generalization is an illusion.
An eval that never absorbs new failure modes slowly stops measuring reality. If production surprises never flow back into the dataset, the suite guards yesterday's bugs and misses today's.
Your eval suite is itself a system that needs measuring. These signals tell you whether to trust its verdicts.
| Metric | What it tells you | Good sign |
|---|---|---|
| Judge-human agreement | How closely the judge tracks human labels (correlation / kappa) | High: judge scores can be trusted |
| Failure-mode coverage | What share of known failure types the dataset actually exercises | High: few blind spots left |
| Pass rate / score distribution | The shape of results, not just the mean, across slices | Healthy spread, no hidden cliffs |
| Variance across runs | How much the same input's score wobbles between runs | Low: results are stable and reproducible |
| Regression catch rate | Fraction of real regressions the suite blocks before release | High: the gate actually protects users |
| Eval cost & runtime | Token spend and wall-clock time to run the full suite | Cheap and fast enough to run every PR |
Three representative patterns showing evals and LLM-as-judge in production-style use.
A team ships a RAG assistant and needs proof that answers stay grounded in the retrieved passages instead of inventing facts.
A support chatbot is edited constantly - new prompts, new tools, new model versions - and each change risks silently breaking prior behavior.
An engineer has two candidate system prompts and needs a defensible answer to "which one is actually better?"
Evals are moving from occasional spreadsheets to continuous, automated infrastructure that sits at the center of every LLM product.
Scoring not just final answers but the whole tool-using trajectory - did the agent take the right steps, call the right tools, and recover from errors?
Judges that continuously fit themselves to fresh human labels, reporting their own agreement so you know exactly how far to trust them.
Models that generate targeted edge cases and adversarial inputs to grow coverage far beyond what humans can curate by hand.
Position, verbosity, and self-preference controls baked into judge frameworks - swapped orders, length normalization, cross-family graders.
Reference-free scoring on live traffic becomes standard telemetry, catching drift the moment it appears rather than at the next release.
Dedicated suites for jailbreaks, toxicity, PII leakage, and policy compliance shipping alongside quality metrics as a release requirement.