07. Optimization and the Intervention Ladder

1. Purpose and when to read this#

Two proposals land in the same week, against the same measured gap.

The first costs an afternoon. Take twenty failing cases, print what each tool call actually returned, and compare it against the contract the model was told to expect.

The second costs a few weeks and a few hundred priced GPU-hours. Assemble training data, fine-tune, and take on an artifact that has to be rebuilt every time the base model underneath it moves.

Both are competent engineering, and nothing in the failing cases says which one is right. What settles it is whether the cheap check has been run. Because if the tool has been returning its results in a different order than its contract advertises, the fine-tune will still look like it worked — it will learn to cope with the mismatch — and the project now owns a permanent, expensive workaround for a bug the afternoon would have found.

Ordering the available fixes by what they cost, and requiring evidence before each step toward the expensive end, is what this chapter is for. Chapter 00 states that ordering as principle P3. This is its operational form: what each rung costs, what licenses starting work there, what licenses leaving, and what to do when the evidence runs out.

One thing to hold from the first page. The ladder is ordered by cost, cheapest diagnosis first. It is not a ranking of sophistication, and — apart from two specific edges — it is not an empirically validated sequence either. The evidence label in §4 says exactly which parts external evidence supports and which parts are this playbook's own argument. It is worth reading before quoting the ladder at anyone.

This chapter starts after the diagnosis. Chapter 03's taxonomy has already named the root-cause class (RC-1…RC-12), or chapter 06's performance work has already shown that a capacity ceiling is binding. What is left is the choice of what to change next — with its price attached, and with the evidence that justifies paying it, before you pay it.

Read this chapter when:

• A measured gap has a diagnosed root-cause class and you need to pick the intervention.
• An expensive experiment — a training run, a larger model, an architectural change — is being proposed, and you need to know whether it has earned the right to run yet.
• You are setting a generation or reasoning budget, deciding whether to quantize, or choosing between prompt and workflow variants.

Do not read this chapter to find a root cause. That is chapter 03's job. This chapter assumes the diagnosis is already in hand and answers one question: given it, what next, at what cost, on what evidence.

2. Inputs required#

Five things. The first two are hard prerequisites — without them, nothing below has anything to work on.

• A root-cause class from the canonical failure taxonomy (RC-1…RC-12), not a symptom description. "Weak on multi-step questions" is a symptom. If a symptom is all you have, go back to chapter 03.
• A frozen execution-system identity (chapter 02). Every rung below compares a before against an after, and that comparison means nothing if the thing being measured changed underneath it.
• Your project's stakes tier (chapters 00 and 01). It sets how much pre-registration and record-keeping the diagnostic and adoption procedures here demand of you.
• A generation-length sample from the iterate split, if you are calibrating a budget (§5.3).
• A draft hypothesis and a small paired sample, if you are gating an expensive experiment (§5.2): whatever that experiment assumes, however informally written down, plus access to a handful of cases from the population it would run on.

3. Decisions this chapter supports#

• Which rung of the ladder to work at next, given a diagnosis.
• Whether an expensive experiment — training, a larger or different model, an architectural change — has earned the right to run, before it runs.
• What generation or reasoning budget to set, and what to do when truncation still bounds outcomes at a larger budget.
• Whether to quantize an artifact, to what precision, and what evidence a quantized artifact must clear before it ships.
• When to stop optimizing at a rung, and whether stopping means accepting the current gap or descending further.

4. Normative principles#

Four principles. Read each parenthetical label as carefully as the rule it qualifies — the first one carries a scope note that changes what you are entitled to claim for the ladder.

The intervention ladder is ten rungs, cheapest first:

0 instrument integrity  →  1 infrastructure/runtime  →  2 evidence/retrieval/context
→  3 tool contracts & output enforcement  →  4 specification & verification
→  5 generation/reasoning budget  →  6 routing/escalation  →  7 fine-tuning
→  8 larger/different model  →  9 architectural redesign

This is chapter 00's P3, restated for the job this chapter does: the diagnosis picks the rung. Not habit, not what worked on the last project, not what is currently fashionable. §5.1 gives each rung its operational form; §7 states the gate you have to clear to descend past one.

The scope note in the label is load-bearing, and this ladder gets quoted often enough that misstating it matters. External evidence supports two things: rung 0's primacy, and the finding that retrieval and prompting should be tried before fine-tuning on knowledge and citation tasks, where three converging studies agree [EXT-FT-001]. Everything else in the order is a cost argument — rule out the cheap, fast-to-diagnose causes before assuming an expensive, slow-to-diagnose one. That is a defensible way to spend a budget. It is not a measured result, and presenting it as one would be exactly the overclaim this playbook's labels exist to prevent.

Which is why the order is not uniformly binding. §5.1 permits rungs 1–6 to be re-sequenced locally when a diagnosis clearly names a cause class out of order. Rung 0's priority and the never-train-around-defects rule are not locally overridable.

Before committing to an expensive experiment, isolate the cheapest observable that would tell you its core assumption is false, and measure that first.

The first-principles argument is about what the money buys. An expensive experiment rests on an assumed mechanism — the model fails these because it runs out of room to think, say. That mechanism is almost always testable on a small paired sample at a fraction of the cost. Running the expensive version first spends the larger budget to learn something the small one would have told you for free.

The corroboration is weaker than the argument, and is labelled as such. Vendor guidance to "start lightweight, invest early in evaluation, and layer in training-based techniques where measurement shows they're needed" [NV-AGENTICBLOGS-001] is a framing statement, not a study. And the worked case in [SCENARIO: SCENARIO-11] shows a diagnostic run at a small fraction of an expensive experiment's cost correctly gating the decision to proceed — a single instance, not a base rate. §5.2 gives the design pattern.

Stated on its own because it is the single most common shortcut this chapter exists to block.

A capability gap that is really a broken instrument, a serialization defect, an unreachable fact, a missing tool contract, or a task nobody ever specified will not be fixed by fine-tuning. What fine-tuning does instead is teach the model to compensate for the defect. That burns the most expensive rung on a problem a cheaper rung owned, and the workaround has to be re-paid every time the model is retrained. Corroborated by the retrieval- and prompt-first literature for knowledge tasks [EXT-FT-001], and by vendor guidance placing evaluation and lightweight methods before training-based ones [NV-AGENTICBLOGS-001].

A generation that finishes without truncating is not thereby a generation that is right. Only the deterministic scorer — or a calibrated judge, chapter 03 — gets to say that.

This needs saying because completion is a perfectly good proxy right up until the moment you use it for this. Under a fixed budget, finishing and being correct move together. Start optimizing the budget itself and the thing that was holding that correlation in place is precisely the thing you changed. It is the classic failure of optimizing a proxy instead of the target. [SCENARIO: SCENARIO-06] walks through a generation-cap confound inflating a model-comparison headline until the budget factor was isolated, and the direct anti-pattern — letting a completion proxy alone justify a go decision — recurs often enough to name explicitly (§9).

5. Default procedure#

5.1 The ladder, operationally#

Rung 0 comes first. On every gap, every time.

Every other rung asks you to act on a number. Rung 0 asks whether the number is real. Until the integrity gates are green — a measured reachability ceiling showing the items can be passed at all, no inversions, the gold-answer gate passed, determinism verified — you do not have a measured gap. You have a reading.

Work at any lower rung on an unvalidated reading is optimization against an unknown. If the reading was wrong, the work is wasted whether or not the score moves, and nothing in the score tells you which of those happened.

Rung 0 costs hours. Nothing is trained and nothing is bought. It is the cheapest rung on the ladder and the only one that can invalidate every other rung's result, which is why it is entered first even when you are confident the instrument is fine — and especially then.

Now the ladder in full. Chapter 03 defines the twelve root-cause classes normatively; the table below is their operational form — what each rung fixes, roughly what it costs, what to check first before spending at that rung, and two separate evidence bars that must never be conflated:

• Entry evidence (this rung is indicated) — what licenses you to start work at the rung. It is a diagnosis, not an achievement.
• Exit evidence (this rung is exhausted) — what licenses you to stop work at the rung and descend past it. This is the column the descent gate in §7 binds on.

Keeping them apart matters because they get confused in one particular direction. A rung whose entry evidence is absent is inapplicable, not exhausted — if your system has exactly one tier, there is no routing decision to make, so rung 6 was never indicated. Inapplicability satisfies the exit requirement only when it is recorded as a finding, with the evidence that established it. A rung is never cleared by silence.

Cost class (illustrative ordinal — calibrate the actual units to your own accounting, §6):

Transport-vs-contract discriminator (RC-2 vs RC-4). Corrupted bytes can land at one of two rungs, and which one depends entirely on where the corruption starts. If the serving stack, a proxy, or the serialization layer alters bytes independently of any tool, that is RC-2 and belongs at rung 1 — the shape [SCENARIO: SCENARIO-08] teaches. If the payload a tool emits does not match its declared schema, addressing, ordering, or permission contract, that is RC-4 and belongs at rung 3. The discriminating test is cheap: replay the identical payload through the transport with the tool held fixed. If the bytes change, it is rung 1's.

Local re-sequencing. Rungs 1–6 MAY be worked out of order when a diagnosis clearly identifies the cause class out of order — chapter 00 P3's override provision. Rung 0's priority and the never-train-around-defects rule are not locally overridable without a written method decision record. And re-sequencing changes only the order in which rungs are worked. It waives no rung's exit evidence.

A dominant cost or time bucket at any rung is a finding, not automatically waste (defined normatively in chapter 06, applied here). Before treating a rung's expense as overhead to cut, ask whether it is task-intrinsic and whether it moves the decision the ladder work serves. If it is and it does, it is the object of study — route it to the diagnostic gate below, not to an optimization backlog.

5.2 Diagnostic gates: deciding before you spend#

A diagnostic gate is a cheap, pre-registered probe whose only job is to decide an expensive experiment's fate before that experiment runs. It is the mechanism that turns "earn the cost before you pay it" (§4) from an aspiration into something with a procedure. Five steps.

1. Name the hypothesis the expensive experiment assumes. State it as a claim that could turn out false — "additional generation budget converts truncated failures into passes" — rather than as an intention, like "let's try a bigger budget and see." An intention has no result that would stop it.

2. Find the cheapest observable that discriminates it. Not a proxy that merely correlates with success under the current regime (§4). The actual decision-relevant signal. For a budget question, that is pass rate on the deterministic scorer, not completion rate.

3. Pre-register decision bands — GO / RE-SCOPE / DROP — with data-sufficiency precedence, before running anything. Fix a minimum sample-sufficiency floor (n_min). Size n_min from the interval width needed to separate the bands, not from a bare count: the smallest n at which the primary metric's 95% interval, evaluated at the effect you expect, sits wholly on one side of the GO threshold (chapter 04's MDE machinery; §8 works the arithmetic). A floor chosen for convenience produces a design that cannot resolve its own bands — which is the sub-MDE-promotion anti-pattern (chapter 00 §9) wearing a pre-registration's clothes. The precedence rule, applied in order:

   if reconfirmed_sample < n_min:
       verdict = INCONCLUSIVE / RE-SCOPE, regardless of the point estimate.
       (An underpowered diagnostic tells you the premise is unmeasured,
        not that it is false — it can never license DROP, and a favorable
        point estimate below n_min can never license GO either (§9).)
   else:
       apply the pre-registered effect-size bands (GO / RE-SCOPE / DROP),
       reading each band against the metric's INTERVAL, not its point estimate.
       A band the interval straddles yields INCONCLUSIVE / RE-SCOPE.
   

   Always report the interval alongside the point estimate, and read the bands against it. A diagnostic gate that licenses spend is an inferential device, and it is held to chapter 04's standard. A project MAY instead run the gate deliberately as a screening device — cheaper, and legitimate for ordering candidate interventions — but then it MUST print its verdict as RANKED under chapter 04's screening-vs-inference rule, still report the interval, and attach no inferential claim to the result.

   Completion alone never produces GO. A completion-only metric may appear as diagnostic context, but only the primary, pre-registered discriminating metric can license a GO — §4's completion-is-not-correctness principle, operationalized.

4. Run it paired and controlled. Use a contemporaneous paired control: both arms in the same session or window, everything but the intervention held fixed, order pre-registered and counterbalanced (round-robin ordering or an explicit AB/BA assignment). This is required whenever the comparison would otherwise cross a measured reproducibility boundary (chapter 02). A historical-vs-new comparison cannot carry a causal claim across an unmeasured instability boundary — the drift and the effect arrive mixed together, and no arithmetic afterwards separates them.

5. The diagnostic's verdict is the recommendation that binds, absent a documented override. At stakes tiers 2+, run the diagnostic as a diagnostic run kind — ledgered inside the fail-closed record, cryptographically non-promotable (chapter 04's default; the relaxed lane that runs a diagnostic outside the record is the anti-pattern the record exists to prevent, chapter 00 §9). The bands are judgment bands, not a hypothesis test with automatic execution. A human decision owner reads the result and, absent a written override, follows the pre-registered consequence — exactly the discipline a consequence-bearing tolerance enforces (chapter 04) [SCENARIO: SCENARIO-01].

The pattern generalizes directly, and [SCENARIO: SCENARIO-11] is one instance of it: a cheap paired probe with pre-registered bands and a sufficiency floor taking precedence over the primary metric, deciding a much larger experiment's fate before it was allowed to run. Read it for the pattern — and then apply step 3's interval requirement to your own probe. Whether any given probe separates its bands is a property of that probe's n and observed rate. It has to be computed. It is never inherited.

5.3 Generation/reasoning-budget calibration (rung 5)#

What follows is a procedure, not a number to copy. A cap calibrated on one model is not valid evidence for another: output-length distributions shift across model families and versions (§6), which is the shape [SCENARIO: SCENARIO-06] takes apart.

1. Pilot the generation-length distribution on the iterate split. Record the full distribution of tokens actually generated per case, not just the mean.
2. Set the cap above a pre-registered quantile (illustrative default: p99) with explicit headroom. State the quantile and the headroom multiplier in the experiment contract. Both are project **[PARAMETER]**s, calibrated per §6.
3. Define a truncation-rate tolerance with a named consequence — ABORT / RECALIBRATE / PROCEED-WITH-DECLARED-CEILING (chapter 04). Not a bare "we'll keep an eye on it": a tolerance with no named consequence is the unpriced-escape-hatch anti-pattern (chapter 00 §9) applied to budgets.
4. Persist truncation telemetry — finish reasons, per-invocation token counts — as part of the telemetry floor (chapter 12), landed before the run you would otherwise have to reconstruct it from.
5. When truncation still bounds outcomes at the chosen cap, run the diagnostic gate (§5.2) rather than assuming. Do not infer that a larger cap will convert truncations into passes. Measure it on a paired sample before committing to the larger budget as a design decision.
6. State the selection-effect caveat wherever a rescue or conversion rate is reported. Cases that complete inside a smaller budget are systematically the easier ones. So a rescue rate measured at a larger budget is an upper bound on how much of the harder, still-truncated population would convert — not a given [SCENARIO: SCENARIO-06]. §8 states the caveat in the form it has to be reported in.

5.4 Prompt/workflow optimization discipline (rung 4)#

• One factor per arm. A variant that changes the prompt and the output schema and the retrieval call is not a controlled comparison. Whatever it shows, you will not know which change produced it. Isolate the factor under test.
• Enumerate the hypothesis space before generating variants. You SHOULD list the candidate changes, and the mechanism each one is supposed to fix, before writing them. That makes the comparison set a deliberate design rather than accumulated trial-and-error. Once the variant set is larger than a handful, screening and racing discipline applies directly — chapter 04's iterate-split, stratum-balanced rungs [EXT-STOPPING-003].
• Pre-register the adoption rule before seeing results — the margin and the metric that would justify keeping a variant — so that a fluke result cannot retroactively justify itself.
• A variant that changes the execution system is a new identity (chapter 02). Compare it under the same paired-control discipline as any other execution-system change, not as a free-form "try it and see."

5.5 Quantization acceptance gates (a cross-cutting deployment lever)#

Read the status line above as a scope limit on everything in this subsection. This playbook has no internal execution record of a quantization acceptance gate. The recovery-bar shape below is adapted from one vendor's demonstrated practice on its own benchmark suite [NV-QADNEMOTRON-001]. The escalation ladder is inference from that vendor's stated recovery step [NV-MODELOPTQAD-001]. And the statistical binding here is chapter 04's general standard applied to this problem, not a validated quantization-specific procedure. Treat the bar number as a starting reference to calibrate (§6), and the procedure as doctrine to be exercised and then revised.

Quantization does not sit on the capability ladder. It changes the artifact's footprint and cost, not the diagnosis. So it MUST NOT be used to paper over a capability gap diagnosed elsewhere on the ladder, and it MUST be gated by the same evaluation instrument (chapter 03) you use for every other quality claim — never by an academic benchmark that is not your task.

[ADAPT: NV-QADNEMOTRON-001] [ADAPT: NV-MODELOPTQUANT-001] The gate compares the quantized artifact against the higher-precision one it would replace, on your own task evals, as a recovery percentage:

recovery% = task_score(quantized artifact) / task_score(higher-precision baseline) × 100

As-of 2026-08-21, one vendor's demonstrated practice targets >99% median recovery across a benchmark suite before shipping an artifact quantized post-training (PTQ) and nothing else. It treats 95–99% as a deliberate PTQ design target — evidence the artifact was pushed hard enough to bank the size and latency gains — when a further recovery stage is already planned [NV-QADNEMOTRON-001]. That is one vendor's internal practice on its own flagship release, not published doctrine. Adopt the shape of the gate: a stated recovery bar, measured on your evals, with an escalation path below it. Then calibrate the bar itself against your own stakes tier (§6).

The imported number is a different statistic from the one you will compute. This is easy to miss and it makes a loose gate look strict. The vendor's ">99%" is a median across a benchmark suite of per-benchmark recovery ratios — a robust central tendency over many tasks. A project computing recovery on a single aggregate task score is computing one ratio, not the median of many, and a single ratio carries far more sampling noise than the vendor's median does. If you want the vendor's statistic, compute per-stratum recovery ratios and take their median, reporting the spread alongside. If you want a single aggregate, say so, and set the bar against that statistic's own resolvable margin — which is what the rest of this subsection derives.

1. Paired on the same items. Score the quantized artifact and the higher-precision baseline on the identical eval items, in a contemporaneous paired control design. Two separately-collected aggregate scores are not a comparison.
2. Effective N, not raw N. Identify the clustering unit — scenario family, document, session — estimate ICC, and carry N_eff forward. Quantization evals are almost always clustered.
3. MDE stated before the run. Compute the design's MDE on the paired difference at α=.05, power .80 (chapter 04 §8). If the MDE is larger than the accuracy loss you care about, the design cannot answer the question. Enlarge it, or re-scope the bar to a margin the design resolves — before running.
4. Report an interval. State a confidence interval on the recovery ratio, or equivalently on the paired score difference. A bare "97.0%" is not a result.
5. Verdict from the closed set. CONFIRMED (the interval clears the pre-registered bar), REFUTED (the interval falls wholly below it), or INCONCLUSIVE (the interval spans the bar, or the paired difference sits inside the MDE band).

A recovery point estimate inside the MDE band is INCONCLUSIVE, not a ship. This is the specific failure the gate exists to prevent: a 2- or 3-point aggregate difference read as "97% recovery, close enough" by a design that could not have resolved a 7-point difference. INCONCLUSIVE MUST carry a pre-registered consequence like any other tolerance (chapter 04), and the default SHOULD be do not ship the more aggressive artifact — escalate down the ladder below, or enlarge the design. An unresolved accuracy question resolves in favour of the incumbent, not the candidate. §8 works the arithmetic end to end.

Escalation ladder below the bar (each step attempted before the next):

1. Disable KV-cache quantization — the cheapest reversal, recovering accuracy at some memory cost.
2. Partial or mixed precision: quantize fewer layers or tensors.
3. Weight-only quantization alternatives at the same nominal precision.
4. Quantization-aware recovery. As-of 2026-08-21, one vendor names this its recommended strategy for accuracy recovery after quantization [ADAPT: NV-MODELOPTQAD-001].

Format portability trap (chapter 02). A quantization format tied to one GPU generation's numeric hardware support will not execute on a different generation. [PRINCIPLE] Quantize for the deployment target, not the development box. (strong-evidence, [ADAPT: NV-NVFP4PLAYBOOK-001]) If the artifact might ever run on a different hardware generation than the one it was quantized on, choose a portable format even at some efficiency cost, or plan a re-quantization step per target.

Vendor decision-page lag — a contradiction kept visible, with a decision rule. One vendor's own "how do I choose a quantization method" decision page recommends prioritizing an older, higher-precision format first, and does not mention the newer low-precision format at all [NV-MODELOPTQUANT-001]. The same vendor's flagship release demonstrates and recommends that newer format [NV-QADNEMOTRON-001], and the vendor's own recovery guide references it [NV-MODELOPTQAD-001].

What the ledger supports, stated exactly: the flagship demonstration is dated 2026-08-17; the decision page carries no publication date and still omits the format as of the 2026-08-21 verification. The size of the lag is therefore unknown. The page cannot be dated, so it cannot be aged. Both are genuine, current vendor sources; they disagree because they were written for different audiences and are maintained on different cadences.

Decision rule: prefer the most recently dated first-party demonstration — a release blog, a benchmark table, a flagship-model writeup — over an undated or older decision page. Where a page is undated, treat it as unverifiable for currency: not fresh, and not stale by an assumed amount either. Record the as-of date, or the absence of one, for whichever source you follow, in your own method decision record.

6. Project adaptation parameters#

None of the following ships with a universal value. State each one, with its rationale, in the relevant experiment contract or diagnostic pre-registration.

7. Decision gates and stopping conditions#

Reading the entry column as a descent bar is the specific misreading this split exists to prevent, and it inverts the gate. A rung's entry condition being false is exactly the case where descending past it is legitimate — so requiring the entry condition would block the legitimate path and license nothing. "We tried the cheap thing once" satisfies neither column.

For rungs 7–9 the exit bar is the rung's own pre-registered acceptance gate returning CONFIRMED or REFUTED under chapter 04. An INCONCLUSIVE result at rungs 7–9 does not exhaust the rung and does not license the next descent. It says the design could not resolve the question, which is a reason to re-scope the design — never a reason to spend more at the rung below.

reconfirmed sample < n_min?
  ├── yes → INCONCLUSIVE / RE-SCOPE (never DROP; never GO)
  └── no  → apply pre-registered effect-size bands to the metric's INTERVAL
              ├── interval wholly ≥ GO threshold     → proceed to the expensive experiment
              ├── interval straddles a band boundary → INCONCLUSIVE / RE-SCOPE:
              │                                        enlarge the sample, or re-scope
              │                                        the bands to a resolvable margin
              ├── interval wholly inside RE-SCOPE    → re-scope the expensive experiment
              │                                        to what the probe shows converts
              └── interval wholly ≤ DROP threshold   → do not run it; move to the next rung

(a) the rung's exit evidence (§5.1) is satisfied, including a diagnostic gate (§5.2) showing the rung's headroom is exhausted at the current evidence bar; or

(b) the engineering and evaluation cost of continuing at this rung exceeds the value the residual gap represents to the decision the work serves.

The two are not interchangeable. Case (b) licenses stopping work; it does not license descending. Descent still requires (a), or a recorded inapplicability finding, for every rung above the target. And descent past rung 6 to fine-tuning additionally requires the full rung-7 entry bar in §5.1: a taxonomy-identified residual gap, every rung above satisfying its own exit cell (or recorded inapplicable), data that demonstrably contains the skill, and economics that close (chapter 11). Never opportunity cost alone.

Validating §5.5's unexercised doctrine. The quantization gate carries a doctrine-not-yet-exercised marker (§5.5). Here is what would retire it: one full execution — a pre-registered paired comparison with N_eff, MDE, interval and verdict recorded before the artifact is scored; the observed recovery interval and the verdict it produced; whether the escalation ladder's steps recovered accuracy in the order stated; and whether the calibrated bar survived contact with a real deployment decision. Record the result as a case study and revise this subsection from it.

8. Metrics and formulas#

Rescue fraction (the primary diagnostic-gate metric, §5.2):

f_rescue = n_rescue / n_reconfirmed

where n_reconfirmed is the count of primary-population cases that reconfirm the triggering condition under the contemporaneous control, and n_rescue is the count of those that pass the deterministic scorer under the treatment. Undefined, and reported as UNDEFINED (never coerced to 0), when n_reconfirmed = 0.

f_rescue is a proportion, so it carries a proportion's interval. At the small n a cheap diagnostic runs at, use a score (Wilson) interval rather than the Wald form — the formulary's own caveat retires Wald once n·p̂ or n·(1−p̂) falls below about 10 (references/STATISTICS_FORMULAS.md):

center = (p̂ + z²/(2n)) / (1 + z²/n)
hw     = z/(1 + z²/n) × √( p̂(1−p̂)/n + z²/(4n²) )
95% CI = center ± hw

p̂ = f_rescue (proportion); n = n_reconfirmed (count); z = 1.96 at 95%

Worked example (illustrative, invented numbers). A primary population of 25 cases that historically hit a capacity ceiling is re-run under a contemporaneous paired control. 20 of the 25 reconfirm the ceiling in the new session (f_reconfirm = 20/25 = 80%). Of the 20 reconfirmed cases, the treatment arm (raised budget) passes the deterministic scorer on 9: f_rescue = 9/20 = 45%. The pre-registered GO band is f_rescue ≥ 30%.

Read that as a point estimate and it clears the band comfortably. The interval decides, not the point estimate. At p̂=0.45, n=20: center = (0.45 + 0.096)/1.192 = 0.458; hw = (1.96/1.192) × √(0.01238 + 0.00240) = 1.644 × 0.1216 = 0.200 → 95% CI ≈ [25.8%, 65.8%]. The lower bound sits below the 30% GO threshold, so the interval straddles the band boundary: the verdict is INCONCLUSIVE / RE-SCOPE, not GO. The design cannot separate 45% from 30% at n=20 — reading it as GO would promote a sub-resolution margin to a spending decision (chapter 00 §9).

Data sufficiency, computed against the band rather than guessed. Solving the same interval for the smallest n whose lower bound clears 30% at an expected p̂≈0.45 gives n ≈ 40 (at n=40, CI ≈ [30.7%, 60.2%]; at n=35 the lower bound is 29.8% and still fails). So the correct pre-registered floor here is n_min ≈ 40 reconfirmed cases — and note what a convenient round floor would have done: n_min = 12 would have licensed exactly the false GO above. Since the primary population is only 25 cases, this design cannot reach its own floor. The honest moves are to widen the primary population, to re-scope the GO band to a margin 20 cases can resolve, or to declare the probe a screening device and print RANKED with the interval attached and no inferential claim (§5.2 step 3).

Overrides are written, never implied. A decision owner MAY override a pre-registered band, but only via a written override recorded in the diagnostic run's ledger entry (§5.2 step 5). "Absent contradictory evidence, call it GO" is an unpriced escape hatch (chapter 00 §9), not an override.

Selection-effect caveat (mandatory wherever a rescue rate is reported, §5.3 step 6). A rescue rate is a rate on a population, and reporting it without naming that population is how it gets read as something it is not. In the worked example above, the 45% is conditioned on the 20 cases that reconfirmed the ceiling — not on the 25 the expensive experiment would run on, and not on the general population. Cases that stopped reconfirming are a differently-composed subset. Cases that already complete inside the smaller budget are systematically the easier ones. So report f_rescue as a rate on the reconfirmed subset, with the subset's definition attached: it is an upper bound on what the harder, still-truncated remainder would convert, never a population rescue rate. The shape this caveat guards against is the one [SCENARIO: SCENARIO-06] works through.

Quantization recovery (§5.5):

recovery% = task_score(quantized) / task_score(higher-precision baseline) × 100

Worked example (illustrative, invented numbers) — the design check comes first. An eval suite of 400 items, clustered by scenario family (20 families, m=20 items each), measured ICC = 0.10 → DEFF = 1 + (20−1)×0.10 = 2.9 → N_eff = 400/2.9 ≈ 138. Expected paired discordance pd = 0.10. Chapter 04's paired MDE: MDE ≈ (1.96+0.84) × √(0.10/138) ≈ 0.075 — this design resolves a 7.5-point paired difference and no smaller one. That sentence is worth having before any scores exist, because it is what the result will have to be read against.

Now the measurement. Baseline task score 82.0% (328/400); PTQ-only quantized score 79.5% (318/400), scored on the identical items → recovery = 79.5 / 82.0 × 100 ≈ 97.0%. Paired difference = 2.5 points, with SE = √(pd/N_eff) = √(0.10/138) ≈ 0.027 → 95% CI on the difference ≈ [−2.8, +7.8] points, i.e. recovery ≈ [90.5%, 103.4%].

Verdict: INCONCLUSIVE. The 2.5-point observed difference is well inside the design's own 7.5-point MDE, and the recovery interval spans both the 99% PTQ-only-ship bar and the 95% floor of the deliberate-PTQ band. The point estimate "97.0%" looks like it lands neatly inside the 95–99% band; the design cannot actually place it there. The pre-registered INCONCLUSIVE consequence applies: do not ship the more aggressive artifact — take the next escalation step (§5.5), or enlarge the design.

What "enlarge the design" costs, stated so the trade is visible. Resolving a 2.5-point paired difference at pd = 0.10 needs N_eff ≥ pd × ((1.96+0.84)/0.025)² ≈ 0.10 × 112² ≈ 1,254 — about 3,600 items at this DEFF, or the same items redistributed across many more (smaller) clusters, which buys N_eff far more cheaply than adding items inside the existing 20 families. If neither is affordable, the honest response is to change the question: pre-register a bar at a margin the affordable design resolves (here, ~7.5 points ≈ a 91% recovery floor) and accept that finer distinctions are outside this instrument's reach.

Truncation rate (§5.3): the fraction of invocations whose finish reason indicates the generation was cut off by the budget rather than completing naturally — computed per run and tracked against the pre-registered tolerance.

9. Failure modes and anti-patterns#

• Completion-as-success proxy: treating a finished (non-truncated) generation as evidence of correctness, and letting that alone justify a GO or adoption decision, without scoring against ground truth [SCENARIO: SCENARIO-06].
• Training around a defect at rungs 0–4: escalating to fine-tuning or a larger model when the diagnosed root cause is an instrument, infrastructure, evidence, tool-contract, or specification defect. The most expensive rung cannot fix a problem a cheaper one owns.
• Underpowered diagnostic read as GO: a favorable point estimate from a diagnostic gate whose reconfirmed sample is below its pre-registered sufficiency floor, treated as license to proceed anyway.
• Insufficient reconfirmation read as DROP evidence: treating a low reconfirmation rate as proof the intervention does not work, rather than as proof the historical premise is unmeasured in the current session.
• A generation-budget cap copied from another project or an earlier model generation: output-length distributions shift across model families and versions, so a cap not re-derived from the current artifact's own iterate-split distribution is a placeholder wearing a number's clothes [SCENARIO: SCENARIO-06].
• Variant spam: generating many prompt or workflow variants without first enumerating the hypothesis space each is meant to test, or without a pre-registered adoption rule. It turns rung 4 into an unstructured search with no fluke guard.
• Unpriced quantization exceptions: shipping a quantized artifact below its recovery bar without either exhausting the escalation ladder (§5.5) or writing a method decision record stating why the exception is acceptable.
• Recovery percentage as an acceptance test on its own: reading task_score(quantized)/task_score(baseline) as pass or fail with no N, no paired design, no pre-run MDE, and no interval — then shipping (or rejecting) an artifact on a difference the design could never have resolved (§5.5, §8).
• Importing a vendor's median-across-benchmarks bar as a bar on one aggregate score: the two are different statistics with different noise, and swapping them silently makes a loose gate look strict (§5.5).
• Reading the ladder's entry column as a descent bar: treating "this rung is indicated" as "this rung is exhausted" (§5.1). It inverts the gate, since a rung's entry condition being false is precisely when descending past it is legitimate.
• Descending on an INCONCLUSIVE result at rungs 7–9: an unresolved acceptance gate says the design could not answer the question, which is a reason to fix the design, never a licence to spend at the rung below (§7).
• Aging an undated source: assigning a lag or a staleness estimate to vendor material that carries no publication date. Undated is unverifiable for currency — neither fresh, nor stale by any stated amount (§5.5).

10. Vendor recipes#

The quantization acceptance-gate mechanics, the escalation ladder, and the decision-page-lag contradiction are all treated in full in §5.5. This section is the source-attribution summary: what each piece of vendor material supplies, and what it leaves to you.

11. Worked examples#

The scenarios below are invented. Each one illustrates a rule; none of them is the evidence for one. Read them for the shape of the mistake.

• SCENARIO-11 — a cheap paired probe, pre-registered with data-sufficiency precedence, gated an experiment an order of magnitude more expensive than the probe itself.
• SCENARIO-06 — isolating a generation-budget factor shrank a claimed model-vs-model gap substantially, and surfaced the selection-effect caveat that a completion-based rescue estimate is an upper bound, not a given.
• SCENARIO-01 — the general failure mode §5.2 step 5 and §5.3 step 3 exist to close: a detected tolerance breach with no named consequence is not a control.
• See §8 for fully worked illustrative-number calculations of the rescue fraction and the quantization recovery percentage. Both worked examples end in INCONCLUSIVE on purpose: they are the two places in this chapter where a plausible-looking point estimate is most likely to be promoted past what its design can resolve, so the arithmetic that catches it is shown in full.

12. Outputs and artifacts#

• A diagnostic-gate pre-registration, recorded before the probe runs: the hypothesis, the discriminating metric, the decision bands, the sufficiency floor with the interval-width derivation that produced it, the pairing and counterbalancing design, and whether the gate is inferential or a screening device printing RANKED. It lives as a diagnostic run kind inside the fail-closed record (chapters 04, 13).
• A rung-diagnosis log entry: the RC class, the rung selected, and — for every rung above it — the §5.1 exit evidence that cleared it, or the recorded inapplicability finding that excused it. Feeds the record of record (chapter 13) and, where an effect was predicted in advance, the prediction ledger (chapter 04).
• A budget-calibration record: the measured length distribution, the chosen quantile and headroom, the truncation tolerance and its consequence, and confirmation that truncation telemetry is live (chapter 12) before the cap is used in an experiment that depends on it.
• A quantization acceptance-gate report, in two parts. First the pre-run design statement: clustering unit, ICC, DEFF, N_eff, MDE, the pre-registered bar and its INCONCLUSIVE consequence. Then the result: recovery percentage with its interval against your own task evals, the CONFIRMED / REFUTED / INCONCLUSIVE verdict, which escalation step (if any) was needed, and the format-portability check against the deployment target.
• Any principle override, or any vendor-guidance-freshness call, as a method decision record.

13. Sources#

Gap dispositions in this chapter: G14 (finding-vs-waste decision test): COVERED — normatively defined in chapter 06; applied here to a rung's dominant cost bucket (§5.1) and to the stop-optimizing test (§7).