Evals: define and measure useful behavior

Section 01

You cannot ship what you cannot measure

Probabilistic systems do not have a simple pass/fail criterion. The same input can produce different outputs, and some variation is expected. Evaluation is how you decide whether the variation is acceptable — and whether a change improved things or made them worse.

This lesson covers the practical construction of evaluations: datasets, rubrics, metrics, model graders, and the release decisions that depend on them.

Section 02

The inspection station

Think of evaluation as an inspection station on a production line. Products come through, and each is checked against a reference gauge, a set of quality criteria, and occasionally a human inspector's judgement. Products that fail are flagged for review or rejection.

For LLM systems, the products are model outputs. The gauges are deterministic checks (does the JSON parse? does the response contain the required field?). The quality criteria are rubrics scored by model graders or humans. The inspection results determine whether a change is safe to release.

Section 03

Building an evaluation dataset

A versioned evaluation dataset is the foundation. It should cover:

• Representative cases: typical inputs the system will see in production.
• Edge cases: unusual but valid inputs that stress the system.
• High-risk slices: inputs where failure has high consequence.
• Adversarial inputs: attempts to break, jailbreak, or misuse the system.
• Production failures: inputs that caused real failures, captured and versioned.

Section 04

Model graders and calibration

A model grader uses one LLM to score another LLM's output against a rubric. This is necessary for qualities that cannot be checked deterministically (relevance, helpfulness, tone). But model graders require calibration: they have their own biases, failure modes, and sensitivities.

Calibrate model graders against human judgements on a labeled subset. Measure inter-grader agreement. Track grader drift over time and across model versions. A model grader that disagrees with humans 30% of the time is not a grader — it is noise.

Combine deterministic checks, model-graded rubrics, and human review. Deterministic checks catch format violations. Model graders catch quality issues. Human review catches what neither can. All three are needed.

Section 05

Metrics, slices, and release decisions

Aggregate scores hide problems. A 90% pass rate may hide a 0% pass rate on a critical slice. Always report results by slice: by input type, by complexity, by consequence level, by user segment.

Sample size and uncertainty matter. A difference of 85% vs 87% on 100 samples may be noise. Report confidence intervals. Use statistical significance tests. Do not make release decisions on differences within the margin of error.

Release thresholds must reflect the task and its consequences. A chatbot for restaurant recommendations can tolerate more variance than a system that generates legal documents. Document the threshold, the evidence behind it, and the decision-maker.

Section 06

What evaluation cannot do

Evaluation measures performance on the test set, not on production. If your test set does not represent production traffic, your evaluation results do not predict production behavior. Keep your evaluation set current with production patterns.

Evaluation does not guarantee quality. It reduces the probability of shipping a regression. A system that passes all evaluations can still fail in production on inputs the evaluation set did not cover.

Model graders can be gamed. If the system being evaluated learns to exploit the grader's biases (e.g., longer responses score higher), the evaluation becomes a measure of gaming ability, not quality.

Section 07

Unscored lab: design an evaluation suite

Design an evaluation suite for a RAG-based Q&A system.

1. Define 20 evaluation cases: 10 representative, 5 edge cases, 5 adversarial.
2. For each case, define the expected behavior and how to measure it.
3. Choose which checks are deterministic (exact match, JSON schema, citation exists) and which need model grading.
4. Write a rubric for one model-graded criterion. Be specific about what scores 1 vs 3 vs 5.
5. Define release thresholds: what pass rate would you require? By slice?
6. What sample size do you need for the threshold to be statistically meaningful?