Error Analysis: The First Eval You Should Run

Nurture Boss, a property-management AI company, had the problem every team has: the product was clearly wrong sometimes, and nobody could say how often or about what. So they built a simple viewer and read their own conversations. What they found was that the assistant was failing 66% of the time on relative dates — “two weeks from now,” “next Tuesday.” Once they had categorized the failures, written tests for them, and measured, date handling went from a 33% success rate to 95% (A Field Guide to Rapidly Improving AI Products).

No new model. No new framework. They looked at the data, counted what broke, and fixed the thing at the top of the list.

This guide is about that counting step — the one that comes before you write any eval at all. It is the companion to Evaluating Agent Trajectories, Not Just Outputs, which covers how to score an agent run. Error analysis is what tells you which failures are worth scoring in the first place.

TL;DR

Before you write a single eval, sample 30 to 100 traces and read them. Write one plain-language note per trace about the first thing that went wrong. Cluster those notes into a failure taxonomy. Then count.

The counts are the output. They tell you which failures are frequent enough to deserve engineering effort, which are mechanical enough for a cheap code assertion, and which are subjective enough to need a binary LLM judge — and they stop you building the generic “helpfulness” metric that measures nothing your users care about.

Hamel Husain and Shreya Shankar, who teach this method, report spending “60-80% of our development time on error analysis and evaluation.” The two steps have names borrowed from qualitative social science: open coding and axial coding. Neither requires a framework, a vendor, or a GPU.

Why This Has to Come First

The intuitive order is to decide what “good” means, write evals for it, then measure. That order does not survive contact with a real system, and there is a specific research result explaining why.

In Who Validates the Validators? (UIST 2024), Shankar and co-authors name the problem criteria drift: “users need criteria to grade outputs, but grading outputs helps users define criteria.” Their finding is stronger than a workflow preference — “some criteria appears dependent on the specific LLM outputs observed (rather than independent criteria that can be defined a priori).” You cannot write the full rubric up front because reading the outputs is what teaches you what the rubric should contain. Eugene Yan puts the consequence bluntly in AlignEval: “It is impossible to completely determine evaluation criteria prior to human judging of LLM outputs.”

Skipping the reading produces two distinct kinds of waste, and Yan separates them usefully. The first is irrelevant criteria — “generic metrics (e.g., helpfulness) or very low probability defects (e.g., grammar, spelling).” The second is unrealistic criteria that “the technology isn’t ready for.” Either way, “teams squander effort that would have been better invested in evaluating actual defects that occur with moderate frequency.” He also notes where the pressure comes from: out-of-the-box eval suites push generic criteria that are easy to plug in precisely because they ignore the idiosyncrasies of your domain and data.

Arize’s annotation docs compress the whole argument into one sentence worth pinning above your desk: “You can’t automate a check you haven’t defined. Start by reading real traces … spotting failure patterns, and grouping them into a taxonomy. The labels you collect become ground truth - and the taxonomy tells you which evals are worth building.”

And the method is older and more boring than it sounds. Yan frames it as the scientific method in disguise — “It starts with observation aka Look at The Data™” — and warns against the reflex it replaces: “Some folks think that adding another tool, metric, or LLM-as-judge will solve the problems … this sidesteps the core problem.” Husain’s version is the same claim pointed at procurement: “The ‘tools first’ mindset is the most common mistake in AI development,” and error analysis is “the single most valuable activity in AI development and consistently the highest-ROI activity.”

Step 0: Get Traces You Can Actually Read

You cannot read what you did not log. Chip Huyen’s generative AI platform breakdown puts observability — metrics, logs, and traces — in the essential layer for exactly this reason, and Tracing and Observability for Agent Systems covers how to instrument it. Everything below assumes you have complete traces, including inputs, tool calls, and intermediate steps.

The second requirement is a surface where reading them is fast, and here the advice is more opinionated than you might expect. Husain’s position is that “custom data viewers that make it easy to examine AI outputs yield more insights than” generic tooling, and he goes as far as calling a simple data viewer “your most important AI investment.” On How I AI he tells people to make their own annotation tool, and on the Latent Space pod he and Shankar note the high payoff of building your own with something like Lovable or Cursor — a much smaller task now than when the advice was first given.

The argument is not anti-vendor, it is about fit. A generic row-per-output grid makes you scroll to find the one field your domain turns on. A viewer you built shows the customer’s time zone next to the parsed date, because you know that is where the bug lives.

That said, if you already run an eval platform, use what it gives you rather than stalling:

Then there is the question of which traces. Sample deliberately rather than reading whatever is newest or whatever someone complained about loudest — the loud ones are already known, and the interesting failures are usually quiet. If you are already in production, Anthropic’s Demystifying evals for AI agents points at an underused source: “look at your bug tracker and support queue.”

If you are pre-launch with no real traffic, synthetic data is fine to start. Shankar frames the choice as a trap with two jaws: “plan A” is spending months collecting real data before you learn anything, and “plan B” is asking an LLM to generate inputs and never looking at them. The workable path is in between — generate inputs, but have someone with domain knowledge shape the dimensions you generate across, and then actually read the outputs.

Step 1: Open Coding

Open coding is the part that feels too simple to be a technique. You read a trace and write one sentence about what went wrong. That is it.

Two rules make it work.

Write about the first failure, not all of them. The evals FAQ is specific: note “the first failure observed in a trace, as upstream errors can cause downstream issues.” On Lenny’s Podcast, Husain gives it as an instruction — “just write down the first thing that you see that’s wrong, the most upstream error. Don’t worry about all the errors” (22:10) — and on How I AI he is explicit that it is a heuristic for making the work tractable rather than a claim about causality. The effect is large: a trace where the agent misread a date and then made six increasingly confused tool calls produces one note about date parsing, not seven about tool calls. Without the rule, your taxonomy fills with symptoms and the real defect gets one vote.

One person decides. The evals FAQ recommends a single domain expert “who understands your users as your quality decision maker (a ‘benevolent dictator’).” This is about consistency rather than hierarchy: three reviewers with slightly different standards produce a taxonomy that is partly a map of their disagreements. Husain’s LLM-as-judge guide calls the same role the Principal Domain Expert, someone who “will set the standard,” with concrete examples — “A psychologist for a mental health AI assistant. A lawyer for an AI that analyzes legal documents.”

And one anti-rule: do not use an LLM for this step. On the Lenny episode, Shankar’s reasoning is that the model lacks the product context that makes a note useful — it can tell you an output is ungrammatical, but not that this customer is on a plan where that discount does not apply. The notes are where your domain knowledge enters the system, so automating them removes the only ingredient the model does not already have.

The artifact is a spreadsheet, and Husain’s office-hours notes give a minimal set of columns. Four is enough:

The Category column stays empty during open coding. That is deliberate — the moment you start choosing categories, you stop observing and start fitting evidence into buckets you already have. Categories are step two.

While you are here, resolve the question that always comes up: pass/fail rather than a 1-to-5 rating. Husain’s position is that “tracking a bunch of scores on a 1-5 scale is often a sign of a bad eval process,” and the evals FAQ carries a dedicated entry on why binary beats Likert. Yan’s reasoning in AlignEval is the most practical: binary “leads to more accurate data, takes less time, and keeps cognitive load low,” and he points at DoorDash and Llama 2 doing the same. Nobody can tell a 3 from a 4 consistently, including you, including next week.

Step 2: Axial Coding

Axial coding is where the notes become a taxonomy. You read all your open codes together and group them into a small set of failure modes — the clusters that actually recur. The evals FAQ is unambiguous about its importance: “Axial coding is the most important step. At the end, count the number of failures in each category.”

Husain’s definition on the Lenny episode is the one to keep: “axial code basically is just a failure mode … get to these clusters of failure modes and figure out what is the most prevalent” (34:13).

This is the step where an LLM earns its place, because the work is summarisation over text you wrote rather than judgement about your product. Give the model all your notes and ask for a candidate taxonomy, then edit it hard. The prompt does not need to be clever:

Below are free-form notes written by a domain expert, one per trace,
describing the first thing that went wrong in each trace.

Propose 5 to 7 failure categories that cover these notes. For each:
- a short name (2 to 4 words)
- a one-sentence definition that someone else could apply consistently
- two verbatim example notes that belong to it

Rules:
- Categories must be mutually exclusive. If two overlap, merge them.
- Name the defect, not the symptom ("stale inventory data", not "wrong answer").
- Do not invent categories that no note supports.
- Add a final category called "none of the above".

NOTES:
{one note per line}

That last rule is load-bearing. On the Lenny episode, Husain recommends adding a “none of the above” category whenever you use AI to label, because without it the model will force every note into an existing bucket and you will never see that your taxonomy has a hole (43:34). A healthy “none of the above” pile is a signal to add a category, not a sign the labelling failed.

Then edit. The model’s first draft will typically be too granular in one area and too coarse in another, and it will produce at least one category that is really a restatement of “the answer was bad.” Collapse those. You are aiming for a handful of categories a new reviewer could apply to a fresh trace and land on the same answer you would.

The method is mainstream vendor guidance now, not a niche technique. OpenAI’s Cookbook on building resilient prompts with an evaluation flywheel recommends “a two-step method drawn from qualitative research: open coding and axial coding,” suggests starting with around 50 failing traces, and defines axial coding as “the process of identifying relationships between your initial labels.” Same words, same order.

Where the names come from

The vocabulary is borrowed from grounded theory, developed by sociologists Barney Glaser and Anselm Strauss out of what they called the constant comparative method, in Awareness of Dying (1965) and The Discovery of Grounded Theory (1967). In that tradition, axial coding is “the process of relating codes (categories and concepts) to each other, via a combination of inductive and deductive thinking.” Sociologists developed it to build theory from interview transcripts without imposing a hypothesis first, which is structurally the same problem as building an eval suite without imposing a rubric first.

Worth knowing mostly so the terms stop sounding arbitrary — and so you know where to look if you want the full method. (The description above comes from secondary summaries; Strauss and Corbin’s Basics of Qualitative Research is the primary text.)

Step 3: Count, Then Stop

Fill in the Category column, make a pivot table, and sort descending. Husain’s line on How I AI — “counting is always powerful” — is almost embarrassing to quote until you notice how few teams do it. On the Latent Space pod he and Shankar make the point that basic data literacy at the level of pivot tables is genuinely enough for this. No statistics, no framework.

What the counts buy you is an argument. “We should improve date handling” is an opinion. “Date handling is 34% of all first failures, and it is 6 times more common than the tone problem we spent last sprint on” is a decision, and it survives contact with a product manager.

The question people ask next is how many traces. The honest answer is that you stop when you stop learning, and the qualitative-research term for that is theoretical saturation: continue “until new traces stop revealing failure modes.” The evals FAQ recommends at least 100 traces and at least 30 annotated yourself before you reach for help. In practice the number varies a lot — on the Lenny episode, Husain reports people finding saturation at around 40, 60, or even 15 traces depending on how concentrated their failures are (30:29).

Small samples feel wrong to anyone with a statistics background, and Anthropic’s guidance explains why they are defensible here: “large effect size means small sample sizes suffice.” You are not estimating a 2% difference between two prompts. You are finding out that a third of your traces break on the same thing. That signal does not need 10,000 rows.

One extension worth knowing about, because it is newer than most write-ups of this method: the evals FAQ now describes an agent-assisted review loop. After you have annotated around 30 traces by hand, an agent searches the rest for likely instances of the patterns you found, and a human accepts or rejects each one. You keep the judgement and skip some of the scrolling. Do it in that order — the manual pass is what teaches the agent what to look for.

What Changes for Agents

Everything above applies to agents, with three adjustments.

Read the whole transcript, but code only the first failing step. An agent trace is long, and the temptation is either to skim to the end or to annotate every misstep. Neither works. Read enough to understand what happened, then write one note about the earliest point where the run went wrong. For a multi-turn conversation that means reviewing the thread as a unit — LangSmith’s annotation queues support exactly this, since “reviewing an entire thread lets you evaluate a full multi-turn conversation.”

Judge the outcome, not the narration. Anthropic’s vocabulary is worth adopting because it keeps this straight: a task is the thing you ask for, a trial is one attempt, a transcript (also called a trace or trajectory) is the record of what happened, and the outcome is the state of the world afterwards. Their example is deliberately deflating: “the outcome is whether a reservation exists in the environment’s SQL database” — not whether the agent said it made a reservation. Agents are fluent. Fluency is not evidence.

Start from real failures, and keep the set small. Anthropic’s recommendation is that “20-50 simple tasks drawn from real failures is a great start.” That is a much smaller number than most teams expect to need, and it is the same logic as above: you are chasing large effects, and tasks drawn from observed failures are pre-selected for discriminating power.

Once you know which failure modes matter, scoring the run itself is a separate discipline with its own traps — final-answer scoring, process quality versus output quality, milestones and partial credit. That is the subject of Evaluating Agent Trajectories, Not Just Outputs, and this guide deliberately stops at its doorstep. Error analysis tells you what to evaluate; that guide covers how.

One warning from Anthropic that applies to both: “You won’t know if your graders are working well unless you read the transcripts and grades from many trials. At Anthropic, we invested in tooling for viewing eval transcripts and we regularly take the time to read them.” Reading does not stop once you have automated scoring. It is what keeps the automation honest.

From Failure Modes to Evals

Now you have a ranked list. Work down it, and pick the cheapest instrument that can catch each failure.

First, just fix the obvious ones. A surprising share of a first taxonomy is not an eval problem at all — a missing field in a prompt, a tool returning stale rows, a date parsed without a time zone. If you can fix it this afternoon, fix it. An eval that guards a bug you could have deleted is pure overhead.

Then reach for code assertions. Husain’s three levels of evaluation ranks them by cost: “Level 1: Unit Tests, Level 2: Model & Human Eval (this includes debugging), Level 3: A/B testing,” where “the cost of Level 3 > Level 2 > Level 1.” Level 1 assertions “should run fast and cheap,” and the ordering instruction is the one that matters here: “before writing assertions, you should critically examine your traces and failure modes.” Many failure modes are mechanical once you have named them. Quoted a leased unit is a database check. Wrong date anchor is an assertion on a parsed value. These cost nothing to run and never drift.

Use an LLM judge only for the genuinely subjective residue — tone, relevance, whether an explanation actually answers the question. When you do, make it binary. Husain is direct on the Lenny episode: “when it goes to building an LLM as a judge, you need a binary score” (26:42). One judge, one dimension, pass or fail. Build it from your expert’s labels plus their written critiques, which is the core of Husain’s Critique Shadowing technique — the critiques are what let the judge reproduce the reasoning, not just the verdict.

Then validate the judge, and validate it correctly. This is the step most teams skip, and the failure mode is subtle enough to deserve the arithmetic. From the Lenny episode: “if you only have the error 10% of the time, then you can easily have 90% agreement” with a judge that always passes (58:54). A judge that has learned to say “pass” scores 90% agreement on a dataset with 10% failures, and it is worth exactly zero.

So measure on the failure class, not overall. Yan recommends Cohen’s κ, which is “more suitable for binary or categorical data when you want to assess the agreement between raters while adjusting for chance agreement,” and is sceptical of correlation metrics precisely because “they don’t account for chance agreement.” His own practice is to have evaluators return binary outputs scored with classification metrics — precision and recall on the defect class. AlignEval, his labelling app, reports exactly that set: recall, precision, F1, Cohen’s κ, and the raw counts of true and false positives and negatives, and it recommends at least 50 labelled samples before you trust any of them.

How bad is the current state of practice? On the Latent Space pod, Husain estimated off the cuff that around 75% of evals in the wild use LLM-as-judge and roughly 80% of those are not helpful. Treat those as rough figures from a podcast conversation rather than measurements — they are one practitioner’s impression, and they reach us through an episode summary rather than as a direct quote. The directional claim is the useful part, and it is consistent with everything above: most judges get deployed without anyone checking them against a human on the cases that matter.

Make It a Habit

Error analysis degrades into a one-time audit unless you schedule it. The evals FAQ gives a cadence small enough to actually keep: “spend 30 minutes manually reviewing 20-50 LLM outputs whenever you make significant changes.”

The reason it has to recur is that the target moves. Shankar’s data flywheel piece puts it plainly: “humans need to be in the loop for evaluation regularly, as human preferences on LLM outputs change over time.” The taxonomy you built in March describes the failures of the March system. Fix the top three and the distribution reshapes underneath you, which is the point — but it also means your eval suite is now aimed at a system that no longer exists.

Two warning signs that you have drifted back into theatre:

A suite sitting at 100%. The evals FAQ warns against optimizing for high pass rates, and Anthropic states the consequence precisely: “an eval at 100% tracks regressions but provides no signal for improvement.” A green dashboard means your evals have stopped telling you anything. Go read traces again.

Metrics nobody calibrated. OpenAI’s evaluation best practices lists the two anti-patterns that bracket this guide — “vibe-based evals” at one end and “ignoring human feedback: not calibrating your automated metrics against human evals” at the other. Error analysis is the cure for the first; periodic re-validation against human labels is the cure for the second.

Where this habit lives in your release process is a separate question. Failure modes that gate a release belong in the offline suite; the same taxonomy pointed at live traffic becomes your online monitoring vocabulary. Online Evals vs Offline Evals covers that split, and error analysis feeds both — the counts tell you which checks earn a place in each.

Husain’s framing from the end of the Lenny episode is the one that tends to land with non-engineers: “evals are the new PRDs” (1:01:01). A failure taxonomy with counts is a specification of what your product must stop doing, written in the language of what it actually does.

Code: Three Scripts for the Whole Loop

All three are standard library Python, runnable as written. They are deliberately small, because the hard part of this method is the reading, not the tooling.

1. Sample Traces into an Annotation Sheet

Reads traces as JSONL and writes a CSV with the annotation columns already in place, stratified so one high-traffic surface cannot dominate your sample.

# sample_traces.py — stdlib only. Run: python sample_traces.py
from __future__ import annotations

import csv
import json
import random
from collections import defaultdict
from pathlib import Path

RAW = Path("traces.jsonl")
SHEET = Path("annotations.csv")
COLUMNS = ["trace_id", "stratum", "input", "output", "primary_issue", "category", "notes"]


def load(path: Path) -> list[dict]:
    with path.open(encoding="utf-8") as handle:
        return [json.loads(line) for line in handle if line.strip()]


def stratified(traces: list[dict], key: str, total: int, seed: int = 0) -> list[dict]:
    """Take a roughly even number from each stratum, then top up at random.

    Proportional sampling buries the rare surfaces, which is where the
    interesting failures usually are. Even allocation finds them.
    """
    buckets: dict[str, list[dict]] = defaultdict(list)
    for trace in traces:
        buckets[str(trace.get(key, "unknown"))].append(trace)

    rng = random.Random(seed)
    per_bucket = max(1, total // max(1, len(buckets)))
    picked: list[dict] = []
    for name in sorted(buckets):
        rows = buckets[name]
        picked.extend(rng.sample(rows, min(per_bucket, len(rows))))

    remaining = [t for t in traces if t not in picked]
    rng.shuffle(remaining)
    picked.extend(remaining[: max(0, total - len(picked))])
    return picked


def write_sheet(traces: list[dict], key: str, path: Path) -> None:
    with path.open("w", newline="", encoding="utf-8") as handle:
        writer = csv.DictWriter(handle, fieldnames=COLUMNS)
        writer.writeheader()
        for trace in traces:
            writer.writerow({
                "trace_id": trace["id"],
                "stratum": trace.get(key, "unknown"),
                "input": trace.get("input", ""),
                "output": trace.get("output", ""),
                # Left blank on purpose. primary_issue and notes are open
                # coding; category stays empty until axial coding.
                "primary_issue": "",
                "category": "",
                "notes": "",
            })


if __name__ == "__main__":
    if not RAW.exists():
        demo = [
            {"id": f"t_{n:04d}", "surface": ["booking", "billing", "support"][n % 3],
             "input": f"request {n}", "output": f"reply {n}"}
            for n in range(240)
        ]
        RAW.write_text("\n".join(json.dumps(row) for row in demo), encoding="utf-8")

    traces = load(RAW)
    sample = stratified(traces, key="surface", total=100)
    write_sheet(sample, key="surface", path=SHEET)
    print(f"{len(sample)} traces written to {SHEET}. Now go read them.")

The stratification is the only opinionated part. Sampling proportionally to traffic gives you a sheet that is 80% of your busiest surface, and your busiest surface is usually your most debugged one. Even allocation across strata is how the rare, broken path gets into the sample at all.

2. Tally the Failure Modes

Once the Category column is filled in, this produces the table that drives the roadmap.

# tally.py — stdlib only. Run: python tally.py
from __future__ import annotations

import csv
from collections import Counter
from pathlib import Path

SHEET = Path("annotations.csv")
DEMO = Path("annotations.demo.csv")


def tally(path: Path) -> tuple[Counter, Counter, int]:
    by_category: Counter = Counter()
    by_surface: Counter = Counter()
    reviewed = 0

    with path.open(encoding="utf-8") as handle:
        for row in csv.DictReader(handle):
            reviewed += 1
            if not row.get("primary_issue", "").strip():
                continue  # Trace reviewed and found acceptable.
            category = row.get("category", "").strip().lower() or "none of the above"
            by_category[category] += 1
            by_surface[(category, row.get("stratum", "unknown"))] += 1

    return by_category, by_surface, reviewed


def report(by_category: Counter, by_surface: Counter, reviewed: int) -> None:
    failures = sum(by_category.values())
    if not failures:
        print(f"{reviewed} rows, none annotated yet. Open the sheet and "
              "write one note per trace before running this.")
        return

    print(f"{reviewed} traces reviewed, {failures} with a first failure "
          f"({failures / reviewed:.0%})\n")
    print(f"{'failure mode':<30}{'count':>7}{'share':>8}")
    print("-" * 45)
    for category, count in by_category.most_common():
        print(f"{category:<30}{count:>7}{count / failures:>8.0%}")

    top, _ = by_category.most_common(1)[0]
    where = {s: n for (c, s), n in by_surface.items() if c == top}
    print(f"\n'{top}' by surface: " +
          ", ".join(f"{s} {n}" for s, n in sorted(where.items(), key=lambda kv: -kv[1])))

    stragglers = by_category["none of the above"]
    if stragglers > failures * 0.15:
        print(f"\n'none of the above' holds {stragglers} of {failures} notes "
              f"({stragglers / failures:.0%}). The taxonomy has a gap — "
              "read those notes and add a category.")


def write_demo(path: Path) -> None:
    rows = [
        ("t_0001", "booking", "wrong date anchor", "date handling", ""),
        ("t_0002", "booking", "quoted a leased unit", "stale inventory", ""),
        ("t_0003", "billing", "", "", "reviewed, fine"),
        ("t_0004", "support", "lost earlier context", "memory", ""),
        ("t_0005", "booking", "wrong date anchor", "date handling", ""),
        ("t_0006", "support", "odd phrasing, hard to place", "", "unclear"),
        ("t_0007", "booking", "wrong date anchor", "date handling", ""),
        ("t_0008", "billing", "", "", "reviewed, fine"),
    ]
    with path.open("w", newline="", encoding="utf-8") as handle:
        writer = csv.writer(handle)
        writer.writerow(["trace_id", "stratum", "primary_issue", "category", "notes"])
        writer.writerows(rows)


if __name__ == "__main__":
    sheet = SHEET
    if not sheet.exists() or not sum(tally(sheet)[0].values()):
        write_demo(DEMO)  # Stand-in for a sheet a human has filled in.
        sheet = DEMO

    report(*tally(sheet))

Note what the blank-primary_issue rows do: a trace you read and found acceptable still counts in the denominator. Without that, every failure looks like 100% of the problem.

3. Check That Your Judge Is Worth Trusting

Before any judge goes into a pipeline, score it against human labels on the failure class. This reports the numbers AlignEval reports, including the one that catches a judge that has learned to always pass.

# judge_check.py — stdlib only. Run: python judge_check.py
from __future__ import annotations

from dataclasses import dataclass

FAIL, PASS = 1, 0


@dataclass(frozen=True)
class Agreement:
    tp: int   # human FAIL, judge FAIL
    fp: int   # human PASS, judge FAIL
    fn: int   # human FAIL, judge PASS
    tn: int   # human PASS, judge PASS

    @property
    def n(self) -> int:
        return self.tp + self.fp + self.fn + self.tn

    @property
    def recall(self) -> float:
        """Share of real failures the judge caught. The number that matters."""
        return self.tp / (self.tp + self.fn) if (self.tp + self.fn) else float("nan")

    @property
    def precision(self) -> float:
        return self.tp / (self.tp + self.fp) if (self.tp + self.fp) else float("nan")

    @property
    def raw_agreement(self) -> float:
        """Flattering and nearly meaningless when failures are rare."""
        return (self.tp + self.tn) / self.n

    @property
    def kappa(self) -> float:
        """Cohen's kappa: agreement corrected for what chance would give."""
        observed = self.raw_agreement
        p_fail = ((self.tp + self.fn) / self.n) * ((self.tp + self.fp) / self.n)
        p_pass = ((self.tn + self.fp) / self.n) * ((self.tn + self.fn) / self.n)
        expected = p_fail + p_pass
        return (observed - expected) / (1 - expected) if expected != 1 else float("nan")


def compare(human: list[int], judge: list[int]) -> Agreement:
    if len(human) != len(judge):
        raise ValueError("Label lists must be the same length.")
    pairs = list(zip(human, judge))
    return Agreement(
        tp=sum(1 for h, j in pairs if h == FAIL and j == FAIL),
        fp=sum(1 for h, j in pairs if h == PASS and j == FAIL),
        fn=sum(1 for h, j in pairs if h == FAIL and j == PASS),
        tn=sum(1 for h, j in pairs if h == PASS and j == PASS),
    )


def pct(value: float) -> str:
    return "n/a" if value != value else f"{value:.0%}"  # NaN != NaN.


def report(name: str, result: Agreement) -> None:
    print(f"{name}  (n={result.n})")
    print(f"  raw agreement    {pct(result.raw_agreement):>5}   <- ignore this one")
    print(f"  recall on fails  {pct(result.recall):>5}")
    print(f"  precision        {pct(result.precision):>5}")
    print(f"  Cohen's kappa    {result.kappa:>5.2f}")
    if result.kappa < 0.4:
        print("  Not usable. Read the disagreements and rewrite the judge.\n")
    else:
        print("  Usable. Re-check it whenever the product changes.\n")


if __name__ == "__main__":
    # 50 traces, 10% of them genuine failures.
    human = [FAIL] * 5 + [PASS] * 45

    lazy = [PASS] * 50          # A judge that always says "pass".
    real = [FAIL] * 4 + [PASS] + [FAIL] * 3 + [PASS] * 42

    report("judge that always passes", compare(human, lazy))
    report("judge worth shipping", compare(human, real))

Run it and look at the first block. The always-pass judge posts 90% raw agreement, which would sail through most review processes, and a κ of 0 with 0% recall on the failures — the two numbers that tell you it is a constant function wearing a rubric. That gap is the whole reason to measure on the defect class.

FAQ

How is this different from just reading logs when something breaks?

Debugging is reactive and sampled by complaint; error analysis is systematic and sampled deliberately. The difference that matters is the counting. Debugging tells you that a failure exists. Error analysis tells you it is 34% of your first failures, which is what lets you rank it against the other eleven things on your list.

Do I need traces from production to start?

No. Synthetic inputs are a fine starting point, as long as a person with domain knowledge shapes what you generate and then reads the outputs. The failure mode is generating inputs with an LLM and never looking at the results, which gives you the appearance of a dataset without the information. If you are in production, your bug tracker and support queue are already a labelled sample of real failures.

Can I skip the manual pass and have a model do the open coding?

Not the first 30 or so. The notes are where your product context enters the process, and a model does not have it — it can tell you an output reads awkwardly, but not that this customer is on a plan where that offer does not apply. After you have annotated enough to know what the patterns are, the agent-assisted loop in the evals FAQ is a reasonable accelerator: the agent proposes instances, you accept or reject.

How many traces is enough?

Stop when new traces stop revealing new failure modes — theoretical saturation. Published guidance suggests at least 100 reviewed and at least 30 annotated by hand, but the real number depends on how concentrated your failures are, and practitioners report saturating anywhere from 15 to 60. Small samples work here because you are looking for large effects, not small differences.

My taxonomy has twenty categories. Is that wrong?

Usually it means you coded symptoms instead of defects, or you did not apply the first-failure rule. Check whether several categories are downstream consequences of one upstream bug; if the agent misreads a date and then makes four bad tool calls, that is one failure mode, not five. Aim for a handful a new reviewer could apply consistently.

When do I actually need an LLM judge?

Only for failure modes that resist a code assertion — tone, relevance, whether an explanation answers the question asked. Mechanical failures should be caught by cheap deterministic checks, which are faster, free to run, and cannot drift. If you do build a judge, keep it binary and single-dimension, and validate it on the failure class before it gates anything.

Further watching

• Why AI evals are the hottest new skill for product builders — Hamel Husain & Shreya Shankar (Lenny’s Podcast, Sep 2025). The best single resource on this method, and the only one here with verified timestamps. The live Nurture Boss trace walkthrough runs through the first twenty minutes; the method proper starts with the upstream-error rule at 22:10, axial coding at 34:13, and the judge-agreement trap at 58:54.
• Evals, error analysis, and better prompts — Hamel Husain with Claire Vo (How I AI, Oct 2025). The most hands-on version: spend an afternoon reading chats, write one-sentence notes, run a categorisation pass. Also the clearest argument for building your own annotation tool, and includes a segment on analysing agent workflows.
• Evals: How to Improve AI Consistently — Hamel Husain & Shreya Shankar (Latent Space Lightning Pod, Mar 2025, ~27 min). Short and dense. Covers trusting an LLM judge only after checking it against a domain expert, the synthetic-data middle path, and the off-the-cuff 75%/80% estimate discussed above.
• Error Analysis: The AI Engineer’s Best ROI — Hamel Husain & Shreya Shankar (Maven, Mar 2025, 60 min). A free lesson from their paid course, covering the whack-a-mole problem, live-coding an annotation interface, evolving criteria, and why manual annotation precedes LLM judges. Registration required.

Primary Sources

• Hamel Husain and Shreya Shankar, AI Evals: Everything You Need to Know — the canonical procedure, the 30-and-100 trace guidance, theoretical saturation, the 60–80% figure, and the newer agent-assisted review loop. A living document; quotes here are as of its September 2026 update.
• Hamel Husain, A Field Guide to Rapidly Improving AI Products — the Nurture Boss case, bottom-up versus top-down failure identification, and the argument for custom data viewers.
• Hamel Husain, Your AI Product Needs Evals — the three levels of evaluation and their relative cost.
• Hamel Husain, Evals: Doing Error Analysis Before Writing Tests — the minimal spreadsheet template.
• Hamel Husain, Using LLM-as-a-Judge For Evaluation — Critique Shadowing, the Principal Domain Expert, and the case against 1–5 scales.
• Shankar, Zamfirescu-Pereira, Hartmann, Parameswaran and Arawjo, Who Validates the Validators? (UIST 2024) — criteria drift, and the finding that some criteria cannot be defined before seeing outputs.
• Shreya Shankar, Data Flywheels for LLM Applications — why human review has to recur.
• Eugene Yan, AlignEval — aligning AI to human and calibrating human to AI, the two kinds of bad criteria, binary labelling, and the judge metrics to report.
• Eugene Yan, An LLM-as-Judge Won’t Save The Product—Fixing Your Process Will — evals as the scientific method in disguise.
• Eugene Yan, Evaluating the Effectiveness of LLM-Evaluators — Cohen’s κ, chance agreement, and scoring judges as binary classifiers.
• Eugene Yan, Task-Specific LLM Evals that Do & Don’t Work — the task-specific metrics that replace generic ones.
• Anthropic, Demystifying evals for AI agents — task, trial, grader, transcript and outcome; 20–50 tasks from real failures; and why a 100% eval is not telling you anything.
• OpenAI, Evaluation best practices and the Cookbook’s evaluation flywheel — the anti-patterns, and open and axial coding as vendor guidance.
• Chip Huyen, Building A Generative AI Platform — observability as the layer error analysis depends on.
• LangSmith annotation queues, Braintrust human review and Arize AX human annotations — doing this inside tooling you may already run.
• Hamel Husain and Shreya Shankar, AI Evals For Engineers & PMs — the paid course this material comes from, listed for completeness.