03. Evaluation Foundation

1. Purpose and when to read this#

Two teams get the same bad score on the same slice of their evaluation. One of them has a weak model. The other has a test nobody could pass — the facts a correct answer needs were never reachable through the tools the system was handed. The number is identical in both cases and says nothing about which team is which. Both are about to spend a week on prompts.

Only one of those weeks is well spent. Working out which team you are, before you spend it, is what this chapter is for.

The instrument comes first because everything downstream treats it as the arbiter of whether an intervention worked. P1 — evaluate first says no selection, optimization, or training decision is made without a trusted evaluation; this chapter is how an evaluation earns that trust. Read it before selecting a model (05), before optimizing anything (07), and before any claim that a system "got better."

An untrusted evaluation does not fail loudly. It keeps returning plausible-looking numbers while quietly steering the project toward fixing the wrong thing, and it can do that for months without anyone noticing. Two sections are worth returning to rather than remembering: §5.9 every time a suite changes, and §5.7 before adopting any LLM-judge grader.

If you came here for one thing. This is the longest chapter in the book and it is built to be entered in the middle.

2. Inputs required#

Five things have to be in hand before the instrument can be built.

• A stated business decision (chapter 01) and the evidence claim it has to produce. The decision is what determines the effect the suite must be able to resolve.
• Real failure evidence — traces, tickets, logs, transcripts drawn from the actual task population, not a specification written from imagination (§5.1). Categories invented at a whiteboard describe the failures someone expected; the expensive ones are the failures nobody expected.
• The project's stakes tier (00 §6), which sizes how much instrument-validation investment §6 requires.
• A frozen execution-system identity (chapter 02) for anything scored against this instrument. An evaluation is meaningless without knowing what produced the outputs it grades.
• Whatever ground truth already exists — human annotations, a code-driven generator, an authoritative outcome log — and its access constraints.

3. Decisions this chapter supports#

• Whether an evaluation result can be trusted at all, before it is allowed to drive any other decision (P2, the instrument-outranks-the-score gate).
• The shape of the task ontology for this task shape — which of the symptom, root-cause, sanctioned-action, and forbidden-claim layers apply, and what replaces the ones that do not (§5.1).
• Deterministic grading or an LLM judge, and whether a judge's scores mean anything yet (§5.6–§5.7).
• When a suite must be versioned rather than edited in place, and when it must be refreshed (§5.9).
• How much curation a corpus needs against how much volume, and what to do when curation and statistical power pull in opposite directions (§6).

4. Normative principles#

There is a measurement behind the harder half of this. JudgeBench shows a judge two responses to a hard knowledge, reasoning, math, or coding item — one correct, one not — and asks it to pick; a frontier judge scored ~56.6% against a 50% pairwise-chance baseline [EXT-JUDGE-002]. Read the number for what it is: a forced-choice figure showing judges are unreliable at discriminating correctness on hard items. It is not a measure of absolute grading accuracy, and the study did not measure schema conformance, exact/fuzzy match, or execution-result grading.

So the conclusion stands on two legs, not one. Where correctness is hard to discriminate, the judge is measurably weak. Where output is trivially checkable (schema, exact match, execution result), the deterministic check wins on determinism, cost, and auditability without borrowing JudgeBench's number at all. Wherever a deterministic check exists it MUST be preferred; a judge is licensed only for open-ended output, behind the calibration protocol of §5.7. Operationalizes P11.

How much of a score that property can account for is the part worth sitting with. Human-reviewing a widely used benchmark down from 1,699 to 500 tasks moved the measured solve rate on the same models from 16% on the full set to 33.2% on the curated subset [EXT-EVAL-005]. Be exact about what that delta is: it mixes removal of invalid items with a change in the item population, and no published analysis decomposes the two. It therefore MUST NOT be reported as "17 points of instrument noise"; §9's cross-version-subtraction rejection binds this playbook's own citations too. What the episode does establish is the point that matters here: a large share of a measured score can be a property of the corpus rather than the model.

For the decomposed, same-corpus version of that lesson see [SCENARIO: SCENARIO-04], where six flagged forbidden claims were all detector artifacts and the corrected all-pass rate moved from 90.0% to 97.5% with no change to the systems under test. A low score on an unvalidated stratum measures the defect, not the model (P2).

The sequel makes the deeper point: the same benchmark's publisher later reported that even the curated subset carried flawed test cases rejecting functionally correct submissions, and retired it for frontier claims [EXT-EVAL-008] — instrument validation is a standing activity, not a one-time gate.

5. Default procedure#

Eleven steps in roughly the order they are done. §5.1–§5.4 build the instrument. §5.5–§5.8 prove it. §5.9–§5.11 govern it once it exists. Each is written to be readable on its own.

5.1 Deriving a task ontology from a requirement (G1)#

The task ontology is the fixed vocabulary your evaluation domain is allowed to use: what an item can be about, what can have caused it, what may be recommended, and which claims are forbidden. Closed, meaning nothing outside the list is a valid answer. Writing it down is usually the moment a team discovers it did not agree on what it was measuring.

1. Error-analysis-first. Read real traces, tickets, or transcripts from the actual task population before writing a category; take open-ended failure notes, don't start from an imagined taxonomy [EXT-EVAL-003].
2. Synthesize a closed category set, with an explicit symptom/root-cause split where the task shape has one (see the shape map below — on a pure labeling task the two layers are the same object). Every item gets a presenting category and a root cause. Some categories should map to more than one cause on purpose (below).
3. Stratify by root-cause class, not symptom. Sampling, ceiling measurement, and power (chapter 04) all key on root cause; symptom is a display grouping.
4. Define per-category evidence requirements. For every root cause, state which facts a correct answer requires and where they live. This is what makes reachability measurable at all (§5.5).
5. Expect criteria drift; version, don't mutate. The first pass will be wrong in places once real outputs are seen (P6); route corrections through a suite release (§5.9), never a live edit.

For a diagnose-and-act task the output is a four-layer closed vocabulary, versioned as one unit:

The layer set is conditional on task shape; only the forbidden-claim layer is close to universal. The sanctioned-action layer applies only where the system selects an action. The symptom/root-cause split applies only where a presenting signal is genuinely distinct from what is wrong. Place your task on this map before building anything — over-building a diagnostic ontology for a labeling task is a common and expensive mistake:

Whichever shape applies, the vocabulary that is present is closed, versioned as one unit, and changed only through §5.9's checklist.

Illustrative example, carried through this chapter: an IT-helpdesk triage assistant. Symptom cannot log in maps to two root causes at comparable base rates — expired credential and identity-provider outage — disambiguated only by reading account and provider logs. Symptom application crashes on launch maps to one root cause, corrupted local cache, plus one absence case, no defect — could not reproduce.

5.2 The canonical failure taxonomy#

"The system is bad at this" is not a diagnosis anyone can act on. The table below is what replaces it: twelve generic classes, each one naming a kind of repair.

§5.1's root-cause set is project-specific — expired credential means something inside a helpdesk and nothing outside one. The canonical failure taxonomy sits one level up. Every project-specific root cause classifies into exactly one of these twelve generic classes at a time, and so does every instrument or system defect that produces a wrong answer for reasons unrelated to the task domain. A task-ontology root cause answers "what specifically was wrong"; an RC class answers "what kind of fix does this call for."

This is the single generic taxonomy this playbook uses wherever a root cause needs naming — chapters 07 and 14 and the templates consume it unchanged. Two honesty notes govern how the table is read, both stated under it: RC-1 and RC-7 are separated by whose verifier is broken, and RC-10/11/12 are provisional-residual classes rather than ex-ante diagnoses.

Symptom categories are what an operator sees; every diagnosis lands in exactly one of these root-cause classes:

RC-1 vs. RC-7 — the disambiguating rule. Both classes contain the words "verifier" and "grader", and they route to opposite ends of the ladder, so decide deliberately: if the defective component is part of the execution system under test, it is RC-7 (rung 4); if it is part of the measuring instrument, it is RC-1 (rung 0). When in doubt, treat it as RC-1 — the cost of an unnecessary instrument check is bounded; the cost of trusting a broken instrument is not.

RC-1 always ranks first: an unvalidated instrument makes every other class unverifiable. §5.5 is how you validate it before trusting a diagnosis into any other class. The full intervention ladder is chapter 07's operational form; the short normative form is chapter 00 §4 (P3).

RC-10, RC-11, and RC-12 are provisional-residual classes. They are assigned only after RC-1…RC-9 have been excluded, and they are told apart from each other not by an ex-ante test but by the outcome of the rung they select. The operating rule: attempt RC-10 first, because it is the cheaper hypothesis. Treat a rung-7 INCONCLUSIVE or null as the evidence that reclassifies to RC-11. Treat a staged one-variable-at-a-time ladder replay whose residual gap tracks the topology rather than any single component as the evidence for RC-12. (Chapter 07 §5.1 carries the rung-8 and rung-9 evidence bars; chapter 09 §7 carries the post-ablation reclassification back to 07.) Record the provisional class and the evidence that would reclassify it, so the record does not read as a firmer diagnosis than it is.

Stochastic reliability is a cross-cutting property, not a thirteenth class. A system can have the capability and not the repeatability: high pass@1 that collapses at pass^k on identical tasks at the same configuration [EXT-AGENT-001]. That is not diagnosed by adding an RC row. It is diagnosed by measuring pass^k at a pre-registered k (chapter 04) and then asking which class the residual belongs to. Where it maps:

• Rung 1 / RC-2 if the spread is nondeterminism the execution system was not supposed to have (batching, reduction order, an unpinned config) — check this first, because it is a defect, not a policy choice (chapter 02).
• Rung 5 / RC-8 if the fix is generation-time configuration — decoding and sampling policy, self-consistency, or a budget that is too tight to be reliably sufficient. Name the sub-reason (sampling policy vs. budget) in the record; the rung is the same, the fix is not.
• Rung 6 / RC-9 if the fix is a retry, latching, or escalation policy — the system needs a second attempt or a stronger tier, not a different sampler.

If none of those reduces the spread, the capability itself is in question and the provisional-residual rule above applies. A reliability claim is always stated at a pre-registered k; a pass@1 number is not a reliability claim.

Non-ML and legacy incumbents map onto the same classes. The taxonomy names kinds of fix, not model internals, so a rule-based, scripted, or human-process incumbent classifies without inventing new rows. An incomplete or stale rule set is RC-6 — the system was never told. A rule set that fires on the wrong evidence because the data it needs is not available to it is RC-3. A broken integration or a mis-parsed payload is RC-2 or RC-4. A workflow that silently accepts an unverified result is RC-7. Work routed to the wrong queue, or never escalated to a human, is RC-9. And "no rule set could express this" is the legacy analogue of RC-11/RC-12. Diagnose the incumbent with the same instrument you will later use on the replacement — that is the only way the comparison means anything (chapter 01).

5.3 Corpus design#

Six design rules. The first is about representativeness; the next three plant specific things in the corpus so specific bad habits become measurable; the last two are about building it without contaminating it.

• DEFAULT Representative sampling / stratification. (consensus) Sample proportional to (or deliberately oversampled relative to) real-world frequency per §5.1's strata; state which and why.
• DEFAULTDistractor design. (inference — first-principles: an option that is never correct cannot be scored right by name-matching) Keep at least one action or label valid for no item in the corpus. Anything that reaches for it is matching a name that sounded relevant rather than reasoning from evidence — the distractor is what turns that suspicion into a measured rate of name-anchoring. Helpdesk example: a reissue hardware token action sanctioned for no current root cause.
• DEFAULTAbsence case design. (inference — first-principles) Items whose correct answer is "nothing is wrong" — this prices confabulation directly. A corpus where every item has something to find rewards a system that always finds something, and you will not discover that habit until it is running against real inputs.
• DEFAULTForbidden claim design. (consensus) Enumerate assertion classes that would trigger a costly or harmful real-world action if believed, and score them as automatic case failure — not a proportional markdown. Helpdesk example: confirmed the user violated security policy, which is never derivable from login evidence alone.
• DEFAULT Per-instance consistency, not just the modal case. (case-study [SCENARIO: SCENARIO-04]) Every generated instance — not only the typical one — must be checked against the forbidden hypotheses it is supposed to rule out. A generator that is right on average but wrong on a subset silently hands the model a forbidden claim that is true of the world it generated.
• DEFAULT Background content through the real pipeline. (case-study [SCENARIO: SCENARIO-04]; mechanism is first-principles) Filler and background content SHOULD be produced by the same generation path as primary content. A direct-write shortcut carries whatever the real pipeline would have normalized — an ID format, a timestamp pattern, a field-ordering difference — and any such artifact that correlates with a stratum is a model-visible shortcut unrelated to the intended signal. Where a bypass is unavoidable, an explicit artifact check MUST run before the corpus is trusted: can a trivial classifier separate bypassed from pipeline-produced items? A positive result is a structural-leakage finding under §5.8.

Shape-dependence of these bullets. The distractor bullet's action form, and everything keyed to a cause-to-action table, apply only to task shapes that have an action set (§5.1's map). On classification, extraction, and generation shapes the distractor becomes a label, field, or claim valid for no item; the absence case becomes the "no finding / nothing to report" item; and the forbidden-claim layer becomes forbidden fabrication on extraction shapes. Representative stratification, per-instance consistency, and the real-pipeline rule apply on every shape that uses a generator at all. For human-annotated corpora, see §6's non-generator procedure.

5.4 Ground-truth design#

Three places a known-correct answer can come from, and what each one does not give you:

The industry-consensus loop — objective → dataset → metrics → run → iterate, with the explicit instruction to validate model-graders against humans before trusting them — is broad practice [EXT-EVAL-001], [EXT-EVAL-002](methodology citable; the hosted platform behind [EXT-EVAL-002] sunsets 2026-11 per its own notice). §5.7's protocol is that validation, made explicit.

Gold-label boundary (restated from §4). Gold labels score outcomes and may fit pre-registered calibration on the iterate split; they are never readable by the system under test, and never select/route/tune anything on qualify/confirm splits. Enforcement mechanics live in chapter 13; split-statistics mechanics in chapter 04. This is G16's boundary rule; §7's gate enforces it operationally.

5.5 Instrument validation#

If you act on only one section of this chapter, make it this one.

The methods below are written for a code-driven generator — reachability is replayed, determinism is checked by regenerating, gold answers are recomputed. If your ground truth is human-annotated instead, do not skip this section: read the Catches column as the checklist you still owe, and run §6's non-generator procedure, which supplies a counterpart for every row.

Three checks catch most instrument defects, and none of them needs a spare week.

Is this stratum winnable at all? Take the stratum with the worst score and work one of its items the way the system has to work it: same tools, same evidence, no glance at the answer key. A stratum that defeats a determined human defeats every model, and its score is a fact about your corpus. Systematized, this is the reachability ceiling — replay the stratum's evidence plan against the corpus you actually built, through the real tool surface, and count the items where the whole required evidence set came back (§8.1). Never argue a ceiling from the generator's intent; measure it against what the tools return.

Did the ordering come out backwards? Rank your systems by how strong you believe they are, then read each stratum's scores in that order. Any stratum where a weaker system wins is a bug report, not a finding: weak-beats-strong means the stratum rewards guessing or something in the instrument is broken there. That is the inversion check, and it costs one sort.

Where do the arms disagree? Pull every item two systems answered differently, and have a person read it before either arm is credited or blamed. This is the check that catches scorers and harnesses, because a scorer defect usually surfaces as a disagreement that makes no sense rather than as a low average. Two patterns deserve special suspicion: a result that is surprisingly good, and an item every system fails in exactly the same way.

All three read the suite as a whole rather than any single item, which is why they need breadth — many items, several strata, more than one system under test. A corpus built that way can audit itself.

The full check list. The three above plus four that are cheap, mechanical, and belong in your release gates rather than in an afternoon of investigation:

Static integrity gates run all of the above on the full corpus, as executable commands, and MUST be protected from subsetting — they are the instruments that catch instrument defects, so sampling them means sampling exactly where a defect would then sit undetected. [SCENARIO: SCENARIO-04] found thirteen harness/scenario defects this way, every one initially indistinguishable from model weakness.

5.6 Deterministic graders vs. LLM judges#

Grader design is chosen by task shape, not by convenience [EXT-EVAL-001]:

The boundary in the mixed row is the one that matters in practice. A deterministic verifier checks objectively-checkable properties — citations resolve, IDs valid, values in-range, action codes sanctioned, schema conformance — and should not be pushed into judging reasoning nuance. Where it starts to strain is exactly where a calibrated judge belongs, and where a stricter deterministic rule does not.

5.7 The judge-calibration protocol#

An uncalibrated judge is an opinion delivered in a format that looks like a measurement. Turning it into a measurement means showing two things: that it agrees with people who know the answer, by more than chance alone would produce, and that it is not being moved by properties of the output that have nothing to do with quality.

A judge score MUST NOT drive any decision until all eight of the following are complete and reported:

1. Sampling design and anchor-set reliability. Take a stratified sample against a human-anchor set — labels the judge is compared to, not trained on. The anchor set MUST be dual-labeled independently, with disagreements adjudicated by a documented rule (third annotator, or a written tie-break procedure), and the human–human chance-corrected agreement MUST be reported alongside κ(judge, anchor). State which label form κ is computed against — individual annotator labels or the adjudicated consensus — because it changes the number.

   Both numbers are needed because κ(judge, anchor) is bounded above by the anchors' own reliability, so it is interpreted relative to that ceiling: κ = 0.55 against anchors whose human–human κ is 0.60 is a different claim from κ = 0.55 against adjudicated anchors at 0.90. An anchor set whose own agreement is below the pre-registered judge bar (step 8) cannot license any gate outcome — fix the annotation guideline first.

2. Chance-corrected agreement, not raw percent. Two raters who both label almost everything the same way will agree most of the time without either of them knowing anything. The reason to correct for that is arithmetic, not empirical: κ subtracts the agreement expected from the raters' marginals, so whenever labels are unbalanced raw agreement exceeds κ by construction (§8.3's formula makes this visible). Report the right estimator: Cohen's κ for two raters with nominal categories; weighted κ (linear or quadratic — state which) for two raters with ordinal categories; Fleiss' κ for three or more raters. Corroboration, not warrant: a single recent multi-judge study (contested, REFERENCE — treat as illustration, not law) reports 33–41 percentage-point deflations between raw agreement and κ [EXT-JUDGE-003].

3. Bias audits — each a measured quantity with a pre-registered tolerance. A named bias with no number and no consequence is not a control (P7). Before scoring, declare each applicable audit's metric, its tolerance, and the consequence of a breach, as a consequence-bearing tolerance:

   Audit	Measured as	Applies to	On breach
   Position bias	Verdict-flip rate under order swap on a paired subsample (same two responses, both orders)	Pairwise judges only — N/A for single-output scoring, mark it so	Judge not usable until the prompt/protocol is changed and re-audited
   Length/verbosity preference	Correlation (or regression slope) of judge score against output length across items of equal anchor-rated quality	All judges	As above; a nonzero slope within tolerance is reported as a declared ceiling on the judge's scores
   Self-preference	Score differential on outputs from the judge's own model family vs. others, matched on anchor-rated quality	Any judge scoring output from its own family	Exclude same-family arms from judge-graded comparison, or change judge
   Style bias	Score differential across pre-registered surface-style variants of identical content	All judges	As position bias

   Arm blinding is mandatory and separate from the audits. The judge's input MUST NOT identify which arm produced an output. Any identifying artifact — formatting, IDs, headers, a model's self-reference — is a structural-leakage finding under §5.8, not a style quirk.

4. Rubric stability across paraphrase. Say the same thing several different ways; a rubric worth trusting returns the same verdict each time. Measured as the verdict-flip rate over k pre-registered paraphrases of the same answer (state k and how the paraphrases were produced, before scoring), against a pre-registered maximum flip rate. Exceeding it means the rubric is under-specified: fix the rubric, then re-run steps 1–4. It is not a licence to pick the paraphrase the judge likes.

5. Domain-transfer check before reuse. A judge calibrated on one domain does not carry its agreement rate to another; strong published agreement (~80% on open-ended chat preference [EXT-JUDGE-001]) is not evidence for a domain with objectively verifiable answers — that domain belongs to §5.6's deterministic default.

6. Drift monitoring and recalibration triggers. Recalibrate on a pre-registered trigger, not on suspicion.

7. Report κ with a confidence interval, every time — never raw agreement alone. CI construction and anchor-set sizing for a target CI half-width are in references/STATISTICS_FORMULAS.md §13; §8.3 has the point-estimate formula and a worked example.

8. Pre-registered κ bar and CI-width ceiling — written before step 1 is executed. It is ordered last because it governs how the other seven are read, but it MUST be committed before any anchor item is scored: state the minimum κ and the maximum acceptable CI width that license each of §7's three gate outcomes, as a consequence-bearing tolerance with the consequence named (§6 has the parameter and tier-scaled guidance). A bar chosen after seeing κ is not a bar; it is a description.

A useful calibration-reference pattern: build a pool of known-good/known-bad judgments with expected scores (a "calibration bag"), and score any new judge configuration by its deviation (z-score) from that baseline rather than an absolute pass/fail line [ADAPT: NV-GARAK-001] — documented for security-probe regression baselines and generalizes directly to judge calibration.

5.8 Leakage and contamination protection#

Leakage is any path by which the test reaches the thing being tested — the held-out content, its answers, or its distribution. Only some of it involves anyone copying a file, which is why the controls below are structural rather than procedural.

• PRINCIPLE Split isolation. (consensus) Disjoint generation seeds/sources between iterate/qualify/confirm splits (mechanics: chapter 04).
• DEFAULT Private held-out sets. (consensus) Never publish qualify/confirm content in full.
• DEFAULT Canary strings. (strong-evidence [EXT-EVAL-007]) A unique marker plus a do-not-train statement in any published excerpt. The marker means nothing on its own, so verbatim reproduction has only one explanation, and that makes leakage detectable rather than arguable.
• DEFAULT Contamination checks. (strong-evidence [EXT-EVAL-006]) Periodically re-test against a freshly collected, decontaminated slice; a material score drop is prior exposure, not regression.
• PRINCIPLE Train/eval separation. (consensus) Training data MUST never touch qualify/confirm content (mechanics: chapter 09; enforcement: chapter 13).
• PRINCIPLE Structural leakage. (case-study [SCENARIO: SCENARIO-03]) A feature letting a learned or judge-mediated component infer which stratum an item belongs to — rather than the signal it is meant to learn — is a leakage vector even with no gold field anywhere in sight. This is the kind that survives careful data hygiene, because nobody copied anything. The audit that catches a router encoding item identity (leakage audit, class-identity ceiling; [SCENARIO: SCENARIO-03]) applies to any grading or gating component.

5.9 Suite versioning and release contracts#

A scorer bug surfaces on a Tuesday. Fixing it in place is an afternoon's work, and it also voids every number already circulated and makes the fix permanently indistinguishable from one chosen because a particular system's answer looked wrong. The licensed alternative is to cut a version.

A suite release is a pre-registered document — written and frozen before any implementation or scoring against the new version — with a fixed skeleton:

Template: templates/EVAL_SUITE_RELEASE_CONTRACT.md. [SCENARIO: SCENARIO-09] used this skeleton to make criteria drift explicit and auditable instead of silent, keeping old-version results as labeled history — never subtracted from new-version results.

A raw score delta across two instrument versions is not a capability measurement. Decompose it: removed false-failures, removed defensible-but-wrong readings, genuinely different evidence presented. None alone is "the model changed" (§9).

Refresh triggers. A suite is due for review when the look ledger (chapter 04) crosses a pre-registered exposure threshold, a static integrity gate fails, or criteria drift is confirmed rather than merely suspected. Refresh follows the same release-contract discipline as any version bump.

Changing the closed vocabulary (checklist). Any change to the root-cause set, action set, or cause→action mapping:

1. Bump the suite/ontology version.
2. Update every synchronized copy — model-facing (if taught in-context) and the scorer's — with an automated equality test, not documentation discipline alone.
3. Re-check reachability for every new/changed class against the built corpus.
4. Record the change and its rationale.
5. Never compare results across versions without saying so.
6. Never change a mapping in response to one system's answers.

5.10 Regression suites vs. capability suites#

Two suites, answering two questions that pull in opposite directions:

Conflating them produces one suite that is too expensive to run often and too noisy to decide with — the worst of both. A fine-tune or intervention gate (chapters 07, 09) runs the full regression suite, never just the target slice: a slice-only gate cannot see forgetting anywhere else.

5.11 The smoke tier#

The smoke tier is a third thing, below both suites in the table above, and it exists for one purpose: catching harness, schema, and tool-contract breakage before real spend.

At Tier 1 a smoke-scale suite MAY stand in for full static integrity gates for direction-finding only (§6's tier table); it never stands in for them at Tier 2+, and never for an adoption claim at any tier.

6. Project adaptation parameters#

Non-generator instruments (G3)#

Most of §5.5's machinery assumes a code-driven generator: reachability is replayed, determinism is checked by regenerating the corpus, the gold-answer gate re-runs computed answers. On a corpus of real, human-annotated material — the normal condition for regulated work, and for every project that arrives without a trusted eval — none of that is executable. This subsection is the substitute, and it is a procedure rather than a caveat: run it in order and every check in §5.5 has a counterpart.

1. Annotation protocol and reliability. Dual-label every item independently; adjudicate disagreements with a third annotator or a documented tie-break rule; report chance-corrected inter-rater agreement (§8.3's κ, with a CI — references/STATISTICS_FORMULAS.md §13) alongside the label set, and state whether downstream scoring uses individual or adjudicated labels. The reported human–human κ is the corpus's own reliability ceiling: no judge, model, or scorer can be credited with agreement above it (§5.7 step 1).
2. Sampled-audit ceiling, sized to the precision the threshold needs. Reachability cannot be replayed against a generator that does not exist, but solvability can be audited. A domain expert attempts each sampled item using only the evidence the system would have, and the fraction solvable is that stratum's ceiling proxy. Size the audit by the precision the decision needs, not by convenience: the ceiling has to be resolvable apart from the stratum's passing threshold, so the audit's margin of error MUST be smaller than the declared threshold-to-ceiling margin (§8.4). Use the binomial margin-of-error table in references/STATISTICS_FORMULAS.md §11 to pick n, and report the ceiling as an interval, never a point.
3. Auditor independence. The expert auditing solvability for a stratum MUST NOT be one of the annotators whose labels are that stratum's gold. An annotator re-attempting their own item measures recall of their own decision, not solvability. Where the expert pool is too small to separate them fully, declare the overlap and treat the ceiling as an upper bound.
4. Gold-answer gate, non-generator form. Replay the adjudicated gold answers through the scorer exactly as §5.5 requires; they MUST score at ceiling. This one check transfers unchanged, and it is the cheapest defense there is against grader-polarity and parsing defects, so run it first each release.
5. Determinism analog: double-annotation drift. There is no corpus digest to compare, so measure the human equivalent: re-annotate a fixed, pre-registered subsample under the current guideline and compare against its stored labels. Material drift is criteria drift (P6) and triggers a suite release (§5.9), not a quiet relabel. Digest what can be digested — the annotation-guideline version, the annotator roster, and the adjudication rule — and record it in the release contract (templates/EVAL_SUITE_RELEASE_CONTRACT.md), so a score is always attributable to a guideline version.
6. Cross-arm disagreement review, non-generator form. Unchanged in intent: every item where arms disagree is reviewed by a human before either arm is credited or blamed. Route the review to an adjudicator, not to whoever proposed the arm under review.
7. Judge-mediated ground truth. Licensed only behind the full protocol of §5.7; an uncalibrated judge is an opinion in a confidence-inspiring format, not evidence. On a human-annotated corpus the judge is calibrated against the adjudicated labels, and its κ is read against the human–human κ from step 1.

Contradiction: volume vs. curation (G20)#

Should a corpus be big or should it be clean? Two positions, both externally sourced, and they do not agree:

• Position A — volume. Prefer a larger corpus at slightly lower per-item signal quality over a small curated one; more items give more power and catch more failure modes even if some are noisy [EXT-EVAL-001].
• Position B — curation. A large, uncurated corpus can undermeasure a system by mixing broken items with real ones. Hand-reviewing a widely used benchmark down to a fraction of its size moved the measured solve rate of the same models from 16% on the full set to 33.2% on the curated subset [EXT-EVAL-005] — a delta that mixes invalid-item removal with a change in the item population and is not decomposed by any published analysis (§4). It shows that corpus composition can dominate a score; it does not quantify how much of a given score is instrument defect.

Decision rule. Neither position wins unconditionally, and there is no formula here — the honest form is a two-factor grid you place your corpus on.

Axis 1 — how items are graded. Deterministically gradable at low marginal cost (generator-derived, schema/exact-match/execution-checked) → volume: more items buy power and failure-mode coverage, and a few noisy items get diluted. Judge-mediated or expensive to verify (human adjudication, clinical or legal review) → curation: a bad item in a small judged set costs proportionally more, and judge noise compounds instrument noise.

Axis 2 — stakes tier. Tier 1 and Tier 2 follow axis 1. Tier 3 curates regardless of grading cost — an unreliable instrument is itself the risk being managed.

The binding quantitative constraint, whichever direction you choose: the smallest stratum must still resolve the MDE its decision needs (the corpus-size parameter below, chapter 04's machinery). That is the floor on shrinking a corpus, and it is what makes "curate harder" a bounded instruction rather than an open licence.

Precedence when curation and power collide (Tier 3). Three forces pull against each other here. "Curate regardless" reduces N. Chapter 04's guidance to tighten α or power at Tier 3 increases the required N_eff. Clustering — by annotator, site, document, or session — inflates DEFF on top of that. The MDE-sizing rule cannot arbitrate between them on its own, because at cold start you do not yet have the ICC and discordance it needs (chapter 04's freeze rule permits a declared prior ICC plus a pre-registered re-estimation procedure — use it). When they collide, exactly three moves are licensed, and each one is written into the contract:

1. Widen the MDE and say so — declare the larger effect the study can resolve and accept INCONCLUSIVE below it.
2. Reduce the number of arms — fewer comparisons on the same labels.
3. Narrow the population — a smaller, well-defined stratum resolved properly beats a broad one resolved not at all.

Silently keeping a tightened α/power on a corpus that cannot support it is not licensed; it produces a study that is under-powered and labeled rigorous, which is worse than an honest wide interval.

Instrument-validation investment by stakes tier#

The same checks at three depths. Find your tier and read across:

Parameters#

7. Decision gates and stopping conditions#

Note the asymmetry: trust keys on the lower bound so that a small, noisy anchor set cannot buy trust it did not earn. The cure for "the CI is too wide" is more anchor items (§6's sizing parameter), never a lower bar.

The one carve-out, stated identically in §6's tier table: an uncalibrated judge may be used as unrecorded triage — deciding which outputs a human looks at next. It may not enter a comparison, an adoption or elimination decision, or any reported number, at any tier. "What should I read next" is not a decision this gate governs; everything downstream of it is.

8. Metrics and formulas#

8.1 Reachability ceiling#

ceiling(s) = reachable(s) / N(s)

reachable(s) counts items in stratum s whose full required-evidence set was actually recovered when the evidence plan was replayed against the built corpus through the real tool surface — never argued from the generator. N(s) is the stratum's item count. In words: the fraction of the stratum a flawless answerer could actually get right.

Worked example (illustrative). A stratum has 40 items; replaying the evidence plan recovers the full required set for 36. ceiling = 36/40 = 0.90. If the stratum's passing threshold is also 0.90, this is a fragile pass — set the threshold comfortably below the ceiling (e.g., ceiling ≥ threshold + 0.05).

8.2 Lookup/shortcut ceiling for ambiguous categories#

This is what §5.1's deliberately ambiguous symptoms buy you, as arithmetic. For a symptom category mapping to k root causes with corpus base rates p_1 … p_k (Σp_i = 1), an uninformed symptom→cause lookup that always guesses the modal cause achieves accuracy max(p_i) on that category.

Worked example. §5.1's helpdesk category cannot log in splits evenly between expired credential and identity-provider outage: lookup accuracy caps at max(0.50, 0.50) = 0.50. A system scoring meaningfully above 50% here is reading evidence, not guessing the modal label.

8.3 Chance-corrected judge agreement (κ)#

κ = (p_o − p_e) / (1 − p_e)

p_o = observed proportion of agreement between judge and human-anchor labels. p_e = proportion of agreement expected by chance given each rater's marginal label distribution. The formula asks how much of the agreement you saw was more than the raters' habits alone would have produced.

Worked example (illustrative). 100 anchor items; judge and human agree on 82 (p_o = 0.82); marginal distributions imply expected chance agreement p_e = 0.55; κ = (0.82 − 0.55) / (1 − 0.55) ≈ 0.60. Report κ with a confidence interval — the CI machinery and the sample size needed for a target CI half-width are in references/STATISTICS_FORMULAS.md §13 — never p_o alone. 0.82 looks strong, 0.60 is only moderate, and the 22-point gap between them is not a surprise: it is what the formula does. Because p_e is subtracted and the remainder rescaled by 1 − p_e, raw agreement exceeds κ by construction whenever labels are unbalanced, and the gap widens as the marginals skew. A single recent multi-judge study (contested, REFERENCE — corroboration, not warrant) reports deflations of 33–41 points in practice [EXT-JUDGE-003].

Read the number against two references, never in isolation: the pre-registered bar (§5.7 step 8) and the anchor set's own human–human κ (§5.7 step 1), which is the ceiling this κ is measured against.

8.4 Recall and passing bar#

recall(item) = |evidence cited that matches required_evidence| / |required_evidence|

A stratum's passing threshold is a chosen bar (illustrative: 0.90) applied per item, then aggregated. The stratum's reachability ceiling (§8.1) MUST exceed this bar with margin, or the bar is unreachable regardless of system quality.

Assumptions box; when these break. All four formulas assume the evidence requirements and root-cause labels are correct (§5.3's per-instance check) and that corpus construction is stable — verified by §5.5's determinism check on a generated corpus, or by §6's double-annotation drift check on a human-annotated one. They measure the instrument's reachability and the judge's fidelity, not the model. An unvalidated evidence requirement inflates a ceiling that isn't real, and a non-representative or unadjudicated anchor set inflates a κ that won't transfer — which is why §5.7 step 1 requires the anchor set's own reliability to be measured and reported. On a human-annotated corpus, §8.1's ceiling is a sampled estimate with a confidence interval (§6 step 2), not an exact count: compare the threshold against the interval's lower bound, not its midpoint.

9. Failure modes and anti-patterns#

• Ceiling-blind scoring: reporting per-stratum accuracy without first computing that stratum's reachability ceiling — an unwinnable stratum reads as a weak model until someone checks [SCENARIO: SCENARIO-04].
• Score-driven harness changes: modifying a scorer, verifier, or rubric because of what a specific system's score looks like, not a proven, independently justified defect (§5.9 checklist step 6).
• Rubric drift from results: widening a rubric because a system disagreed with it, or narrowing it because it now agrees, absent an independently identified defect [SCENARIO: SCENARIO-09].
• Cross-version score subtraction: diffing raw pass rates between two instrument versions and reporting the delta as a capability change (§5.9).
• Silent instrument-version mixing: comparing run rows against manifests from a different suite version without an explicit override — exactly what cross-suite refusal exists to block.
• LLM judge on verifiable output, uncalibrated: substituting a judge for a deterministic check where one exists, or trusting an uncalibrated judge's score for any decision. The deterministic check wins on determinism, cost, and auditability; and where correctness is genuinely hard to discriminate, a frontier judge measured ~56.6% against a 50% pairwise-chance baseline on forced-choice correct-vs-incorrect items [EXT-JUDGE-002](§4 states what that figure covers).
• Judge trust read off a point estimate: declaring a judge trusted because κ's point estimate cleared the bar while its lower confidence bound did not, or choosing the bar after seeing κ (§5.7 step 8, §7).
• A bias "audit" with no number: listing position, length, self-preference, or style bias as checked without a measured quantity, a pre-registered tolerance, and a named consequence — a detection without a consequence is not a control (§5.7 step 3).
• Smoke-tier results quoted as evidence: using a minutes-scale mini-suite's numbers in a comparison, an adoption or elimination decision, or a reported metric. It is a diagnostic run kind by construction and its MDE is enormous (§5.11).
• Pipeline-bypass content generation: writing background/distractor content directly into corpus state instead of through the same generation path, leaving a model-visible artifact unrelated to the intended signal (§5.3).

10. Vendor recipes#

11. Worked examples#

• The helpdesk-triage running example (§5.1, §5.3, §8.2) — a fully synthetic illustration of the four-layer ontology on a diagnose-and-act task: a deliberately ambiguous category, a distractor action, an absence case, and a forbidden claim, with the lookup-ceiling arithmetic worked through. §5.1's shape map says what to build instead when the task is classification, extraction, or generation.
• SCENARIO-04 — thirteen harness/scenario defects, initially indistinguishable from model weakness, found by reachability ceilings, weak-beats-strong inversions, and cross-arm disagreement review.
• SCENARIO-09 — criteria drift made explicit and auditable through a versioned release contract instead of silent in-place edits.
• SCENARIO-03 — structural leakage (class-identity ceiling) caught by leave-one-group-out validation; the same audit generalizes to any learned or judge-mediated grading component (§5.8).
• SYNTH-08 — the human-annotated, judge-mediated path on a Tier-3 project: agreement measured against human reviewers before the judge was trusted, and a pass^k collapse reported instead of the flattering pass^1 number (§5.2's stochastic-reliability note, §5.7, §6's non-generator procedure).
• WALKTHROUGH — the full lifecycle, including instrument construction, on a synthetic non-trivial project.

12. Outputs and artifacts#

• The task ontology itself: a closed, versioned vocabulary carrying whichever of §5.1's layers the task shape requires, version-controlled with the suite, with the shape and any N/A layers stated explicitly.
• templates/EVAL_SUITE_RELEASE_CONTRACT.md — one per suite version, frozen before implementation.
• Static integrity gate results (reachability per stratum, threshold-below-ceiling check, determinism digest or annotation-drift check, gold-answer gate, frontier-saturation check) — persisted per release, not just per run.
• Canary strings embedded in any published excerpt (§5.8).
• A judge calibration report, if any judge is used: the pre-registered κ bar and CI-width ceiling (declared before scoring), measured κ + CI, the anchor set's human–human κ, the bias-audit table with each tolerance and its result, and the domain-transfer check — feeds the prediction ledger and the promotion gates of chapters 04 and 10.
• For human-annotated corpora: the annotation guideline version, annotator roster, adjudication rule, and inter-rater agreement — the non-generator analogue of a corpus digest (§6).
• Input to chapter 05 (candidate selection needs a trusted eval first) and chapter 04 (splits, statistics, and stopping rules operate on the instrument built here).

13. Sources#

Gap dispositions in this chapter:

• G1 (task-ontology derivation from a requirement): COVERED — §5.1.
• G3 (non-generator instruments): COVERED-AS-DOCTRINE-NOT-YET-EXERCISED — §6's seven-step procedure gives every §5.5 check a non-generator counterpart (annotation reliability, sampled-audit ceiling with a sizing rule, auditor independence, gold-answer gate, double-annotation drift, disagreement review, judge-mediated ground truth), but the procedure carries no internal execution record.
• G16 (gold-label usage boundary): COVERED — §4, §5.4 state the boundary rule; enforcement mechanics are chapter 04's, structural isolation chapter 13's.
• G19 (tool-contract design): COVERED for its instrument-validation half only — §5.5, §8.1 (reachability ceilings, evidence-requirement invariants); the tool-contract/schema/addressing half is primary in chapter 08, by design.
• G20 (volume-vs-curation contradiction): COVERED — §6.