PixioLab
Cover of Guide 03, Evaluation Field Guide

Guide 03 · 48 pages · 16 chapters

Evaluation Field Guide

Proving AI systems work, and keeping them working.

By PixioLab

Included in the full program and the guide library. See the 45-day roadmap.

What's inside

  1. 01

    The operating model

    An evaluation is a measurement system, not a vibe check. This chapter gives you the loop every change goes through, the vocabulary the rest of the guide uses, and a map of what to measure.

    Three questions you must be able to answer · The eval loop · The three axes of every eval · Four metric families · Capability evals and regression evals · The maturity ladder

  2. 02

    Error analysis first

    Before you pick a metric, find out how your system actually fails. Reading traces, writing short notes and grouping them into failure categories tells you which evals to build, and which ones you can skip.

    Step 1: Collect traces worth reading · Step 2: Open coding · Step 3: Axial coding · Step 4: Count, rank and decide · Step 5: Repeat until saturation

  3. 03

    Golden datasets and labels

    An eval is only as good as the cases behind it. A golden dataset encodes what "good" means for your product, which is why it is the one evaluation asset you can't buy or copy.

    What a case contains · Coverage and sources · Size, splits and versions · Labels you can trust

  4. 04

    Statistics you actually need

    Every eval score is an estimate from a sample, and often a noisy one. Four tools stop you from shipping noise: confidence intervals, paired comparisons, sample-size planning and non-inferiority margins.

    Where the noise comes from · Report an interval, not a point · Compare versions on the same cases · Repeated trials · Plan the sample size · Non-inferiority: "not worse" needs a margin · Multiple comparisons, overfitting and slices

  5. 05

    Code-based graders and a tiny harness

    When code can check the answer, let code check it. Code-based graders are fast, nearly free and repeatable: build them before reaching for an LLM judge.

    The first question: can code check it? · Metrics for labels and decisions · Checks by task type · A tiny eval harness

  6. 06

    LLM-as-judge

    When code can't check the answer and people can't read every output, a model can grade against your rubric. Done well, it scales expert judgment cheaply; done carelessly, it produces confident numbers that mean nothing.

    Three ways to ask · Rubric design is the actual skill · A binary faithfulness judge · Known judge biases · Cost and latency

  7. 07

    Calibrating the judge

    A judge is a classifier, so measure it like one. This chapter covers how to build a calibration set, which agreement numbers to report, how many labels you need, how to test for bias, and how to correct results for the judge's own errors.

    Build the calibration set · Measure TPR and TNR, not raw agreement · Correct results for judge error · Test the judge for bias · When to recalibrate

  8. 08

    Evaluating RAG

    A RAG answer can break in two places: retrieval can miss the right context, or generation can misuse it. Score only the final answer and you can't tell which one to fix. Measure each surface on its own.

    Retrieval: did we fetch the right context? · Generation: given the context, is the answer good? · The RAG triad · Building a RAG eval set

  9. 09

    Evaluating agents

    An agent takes many steps, calls tools, changes things in the world and can reach the same goal by different paths. Evaluate the outcome, the path and the cost; run every task more than once; and debug from the first wrong step.

    The agent eval stack · Outcome checks: verify the world, not the claim · Trajectory checks: rules over the trace · Metrics that matter · Environments: sandboxes, replay and simulated users · Tracing, and debugging from the first wrong step · Public agent benchmarks, and their limits

  10. 10

    Evaluating models

    Choosing or swapping a model is an evaluation problem. Public benchmarks describe general capability; only your own benchmark tells you how a model will do on your task, at your cost and latency.

    What public benchmarks can and can't tell you · Build a private benchmark, score a scorecard · Quality, cost and latency · Pin versions and watch for change · Evaluating a fine-tune

  11. 11

    Safety evaluations

    Safety evals check that the system stays in bounds when users, documents or attackers push it. Measure attacks by family, measure over-refusal at the same time, and never let a closed hole reopen.

    What to measure · Attack success rate, by family · Always measure both sides · Scoring safety results · A red-teaming method · Agents: judge actions, not just words · Public suites and references

  12. 12

    Regression testing and ship gates

    A regression suite turns your evals into a release check: every prompt, model, retrieval or tool change runs against the same cases, and a gate decides whether it ships. This chapter covers the method; Guide 04 shows the CI wiring.

    Tiers of suite · Hard gates and statistical gates · Living with noise · Baselines

  13. 13

    Evaluating in production

    Offline evals prove a change before it ships. Online evaluation shows how the system behaves on real traffic, catches drift between releases, and feeds new failures back into the golden dataset. It ends with the tool stack that supports all of this.

    Tracing is the foundation · Online signals · A/B tests · Shadow and canary releases · Drift · Guardrails are runtime evaluations · The feedback flywheel · The evaluation stack

  14. 14

    Show the skill

    Evaluation work is easy to claim and hard to fake. Show it with artifacts: a dataset, a calibrated judge, a gate that blocked a bad change, and a report with intervals.

    Talking about it

  15. 15

    Cheat sheet

  16. 16

    Sources and further reading

    Papers, official documentation and tools cited in this guide, grouped by chapter. Start with the first group if you read only three things.