Examples — Scenarios and Worked Examples
Everything in this directory is invented. No real project, client, or measurement appears anywhere in it. What is real is the reasoning: each example works through a decision the playbook has an opinion about, and shows what went wrong when that opinion was ignored.
Read them for the shape of the mistake, not for the numbers.
Scenarios (SCENARIO-01…12)#
Twelve fictional projects, each built around one lesson that is easy to agree with in the abstract and easy to miss in practice. Each follows the same arc — Situation · Decision faced · Evidence · What happened · The generic lesson · What would NOT have worked · References — and ends by telling you what to go check in your own work.
The chapters cite these as [SCENARIO: SCENARIO-NN]. A scenario illustrates a rule;
it is never the evidence for one. Where a chapter claims something is established,
the support is an external source in the ledger, not a
scenario. Chapters stay readable if you skip these entirely.
| ID | Scenario | The lesson in one line | Illustrates chapters |
|---|---|---|---|
| SCENARIO-01 | Consequence-bearing tolerances | A threshold with no named consequence is an escape hatch nobody chose | 00, 04, 07 |
| SCENARIO-02 | Clustered eval, effective N | Count your groups, not your rows | 03, 04 |
| SCENARIO-03 | Learned-router leakage | A learned component must beat the group-identity ceiling, or it learned the wrong thing | 03, 04, 08 |
| SCENARIO-04 | Harness defects | Check the ruler before you trust the measurement | 02, 03 |
| SCENARIO-05 | Hardware purchase discipline | Measure the critical path before you spend the money | 06, 11 |
| SCENARIO-06 | Token-budget confounding | An uncalibrated output cap can invent a quality difference that isn't there | 04, 05, 07 |
| SCENARIO-07 | Deterministic cascade gate | Price routing's ceiling first; a few booleans may capture most of it | 08, 10 |
| SCENARIO-08 | Transport serialization defect | "Transparent" is a claim to test, not a property to assume | 02, 08, 12 |
| SCENARIO-09 | Suite versioning under criteria drift | Version the instrument; refuse comparisons across versions of it | 03, 13 |
| SCENARIO-10 | Pilot optimism collapse | Small-sample headlines run optimistic | 03, 04 |
| SCENARIO-11 | Diagnostic gate | A cheap probe, agreed in advance, can settle an expensive question | 04, 07 |
| SCENARIO-12 | Restart instability, paired controls | Determinism is measured, not assumed | 02, 04 |
The synthetic walkthrough#
WALKTHROUGH_rag-document-qa.md — one complete navigator pass (QUICKSTART → profile → tier → archetype → first sprint → frozen contract) on an invented document-QA project. Entirely fictional, and deliberately shaped unlike everything else here — it exists to show the navigator running end to end on an unfamiliar problem.
Synthetic worked examples (SYNTH-01…10)#
Compact invented scenarios — one per project shape — showing the playbook's rules firing on materially different problems. All names and numbers are invented; each exercises at least one gate or tripwire for real.
| ID | Shape | The point it proves |
|---|---|---|
| SYNTH-01 | API-only greenfield assistant | Eval-first bootstrap; the judge-calibration path for open-ended outputs |
| SYNTH-02 | Privacy-mandated local deployment | Execution-system identity and reproducibility probes before any number is trusted |
| SYNTH-03 | High-throughput extraction | Deterministic grading; operating points; cost-per-successful-task drives model size |
| SYNTH-04 | Tool-calling agent with write actions | Tool contracts, permission gating, injection threat model, observe-only graduation |
| SYNTH-05 | Cost reduction via routing | Oracle analysis and break-even first; a leaky learned router is refused |
| SYNTH-06 | Training rejected | Diagnosis lands on cheaper rungs; the ladder refuses the fine-tune |
| SYNTH-07 | Training justified | The evidence bar actually met: rig-first, ablation, forgetting gate, legality check |
| SYNTH-08 | High-stakes audited deployment | Tier 3: governance at intake, pass^k, human-gated canary, rehearsed rollback |
| SYNTH-09 | An INCONCLUSIVE result | Below-MDE outcomes reported honestly; the decision made on cost, not a fake winner |
| SYNTH-10 | Hardware rent-vs-buy | Demand ledger + break-even + pre-committed trigger; the purchase correctly waits |