
Guide 03 · 48 pages · 16 chapters
Evaluation Field Guide
Proving AI systems work, and keeping them working.
By PixioLab
Included in the full program and the guide library. See the 45-day roadmap.
What's inside
- 01
The operating model
An evaluation is a measurement system, not a vibe check. This chapter gives you the loop every change goes through, the vocabulary the rest of the guide uses, and a map of what to measure.
Three questions you must be able to answer · The eval loop · The three axes of every eval · Four metric families · Capability evals and regression evals · The maturity ladder
- 02
Error analysis first
Before you pick a metric, find out how your system actually fails. Reading traces, writing short notes and grouping them into failure categories tells you which evals to build, and which ones you can skip.
Step 1: Collect traces worth reading · Step 2: Open coding · Step 3: Axial coding · Step 4: Count, rank and decide · Step 5: Repeat until saturation
- 03
Golden datasets and labels
An eval is only as good as the cases behind it. A golden dataset encodes what "good" means for your product, which is why it is the one evaluation asset you can't buy or copy.
What a case contains · Coverage and sources · Size, splits and versions · Labels you can trust
- 04
Statistics you actually need
Every eval score is an estimate from a sample, and often a noisy one. Four tools stop you from shipping noise: confidence intervals, paired comparisons, sample-size planning and non-inferiority margins.
Where the noise comes from · Report an interval, not a point · Compare versions on the same cases · Repeated trials · Plan the sample size · Non-inferiority: "not worse" needs a margin · Multiple comparisons, overfitting and slices
- 05
Code-based graders and a tiny harness
When code can check the answer, let code check it. Code-based graders are fast, nearly free and repeatable: build them before reaching for an LLM judge.
The first question: can code check it? · Metrics for labels and decisions · Checks by task type · A tiny eval harness
- 06
LLM-as-judge
When code can't check the answer and people can't read every output, a model can grade against your rubric. Done well, it scales expert judgment cheaply; done carelessly, it produces confident numbers that mean nothing.
Three ways to ask · Rubric design is the actual skill · A binary faithfulness judge · Known judge biases · Cost and latency
- 07
Calibrating the judge
A judge is a classifier, so measure it like one. This chapter covers how to build a calibration set, which agreement numbers to report, how many labels you need, how to test for bias, and how to correct results for the judge's own errors.
Build the calibration set · Measure TPR and TNR, not raw agreement · Correct results for judge error · Test the judge for bias · When to recalibrate
- 08
Evaluating RAG
A RAG answer can break in two places: retrieval can miss the right context, or generation can misuse it. Score only the final answer and you can't tell which one to fix. Measure each surface on its own.
Retrieval: did we fetch the right context? · Generation: given the context, is the answer good? · The RAG triad · Building a RAG eval set
- 09
Evaluating agents
An agent takes many steps, calls tools, changes things in the world and can reach the same goal by different paths. Evaluate the outcome, the path and the cost; run every task more than once; and debug from the first wrong step.
The agent eval stack · Outcome checks: verify the world, not the claim · Trajectory checks: rules over the trace · Metrics that matter · Environments: sandboxes, replay and simulated users · Tracing, and debugging from the first wrong step · Public agent benchmarks, and their limits
- 10
Evaluating models
Choosing or swapping a model is an evaluation problem. Public benchmarks describe general capability; only your own benchmark tells you how a model will do on your task, at your cost and latency.
What public benchmarks can and can't tell you · Build a private benchmark, score a scorecard · Quality, cost and latency · Pin versions and watch for change · Evaluating a fine-tune
- 11
Safety evaluations
Safety evals check that the system stays in bounds when users, documents or attackers push it. Measure attacks by family, measure over-refusal at the same time, and never let a closed hole reopen.
What to measure · Attack success rate, by family · Always measure both sides · Scoring safety results · A red-teaming method · Agents: judge actions, not just words · Public suites and references
- 12
Regression testing and ship gates
A regression suite turns your evals into a release check: every prompt, model, retrieval or tool change runs against the same cases, and a gate decides whether it ships. This chapter covers the method; Guide 04 shows the CI wiring.
Tiers of suite · Hard gates and statistical gates · Living with noise · Baselines
- 13
Evaluating in production
Offline evals prove a change before it ships. Online evaluation shows how the system behaves on real traffic, catches drift between releases, and feeds new failures back into the golden dataset. It ends with the tool stack that supports all of this.
Tracing is the foundation · Online signals · A/B tests · Shadow and canary releases · Drift · Guardrails are runtime evaluations · The feedback flywheel · The evaluation stack
- 14
Show the skill
Evaluation work is easy to claim and hard to fake. Show it with artifacts: a dataset, a calibrated judge, a gate that blocked a bad change, and a report with intervals.
Talking about it
- 15
Cheat sheet
- 16
Sources and further reading
Papers, official documentation and tools cited in this guide, grouped by chapter. Start with the first group if you read only three things.