12. Observability, Learning, and Promotion

1. Purpose and when to read this#

A long run finished overnight. In the morning somebody asks a fair question about it: did any of those requests get cut off mid-answer, or did every one of them stop because the model was finished?

Which answer you can give was settled before the run started, and it is too late now to change it. Either something wrote a finish reason beside every request while the requests were happening — in which case you answer in about a minute — or nothing did, and the answer is not merely awkward to obtain. It is gone. The requests are over, the process that served them has exited, and no amount of money or cleverness rebuilds a field that nobody wrote.

That asymmetry is what this chapter turns on. Recording the field costs a few bytes and one decision taken in advance. Not recording it costs everything the field would have explained, permanently — and the bill arrives at the moment someone needs the answer, which is always after the only opportunity to pay it cheaply has closed.

Half of what follows is a mature published discipline and half of it is not, and knowing which half is which tells you what you have to build yourself. Production observability — metrics, request tracing, structured logs for a deployed serving stack — is well documented, and vendors ship most of the machinery. The experimentation telemetry that every earlier chapter quietly depends on is not documented anywhere a vendor ships it: the performance autopsy (06), the look ledger (04), and the reproducibility-boundary probes (02) all need records that no product writes for you by default. This chapter supplies the missing minimum: what to instrument before a long or expensive run, the durable record every task execution must leave behind, the discipline that keeps a growing telemetry pipeline honest about its own correctness, and the loop that carries production evidence back into the lab.

One word in the title needs narrowing, because the book uses it for two different things. Chapter 10 promotes a whole system along a deployment pipeline: offline, then shadow, then canary, then rollout. This chapter is about the promotion a single learned or automated component goes through, and it is best read as three positions rather than four stages:

• Observed. The component runs and its decisions are recorded and scored, but nothing downstream acts on them.
• Trusted. It has passed observe-only graduation (08) — the gate that says its recorded decisions were good enough, on evidence that survived a leakage audit — and is now allowed to act.
• Unattended. It goes on acting without a person reading its output. That is a licence rather than a destination, and licences expire.

Chapter 08 issues the licence. This chapter is about the ongoing evidence that keeps it valid and the trigger that takes it back.

Read this chapter:

• Before authorizing any run whose telemetry you would regret not having — a multi-hour experiment, a milestone-defining comparison, anything a future performance autopsy will be asked to explain.
• When standing up production observability for the first time (§5.4, §10).
• When a learned, routed, or automated component has graduated past observe-only (08) and now needs a monitoring plan for as long as it keeps running (§5.7, §7).
• When you are closing the loop from production failures back into the evaluation suite (03) or the training data (09).

2. Inputs required#

• The project's frozen execution system identity (02). Telemetry that is not scoped to a specific runtime, model, host, and configuration is not comparable to anything, including itself a month later.
• The rigor dial and stakes tier from the project profile (00 §6, 01). These set how much of this chapter's artifact set is mandatory for you rather than optional (§6).
• Any existing experiment contract (04) whose runs this telemetry will have to explain. The contract tells you which fields somebody is already counting on.
• The deployment-stage inventory from chapter 10 — offline, shadow, canary, production. The trajectory record carries a deployment-stage field, and that field needs a defined vocabulary to hold.
• Privacy and residency constraints from the project profile (01) and the clean-room boundary (13). These gate what the telemetry floor is permitted to capture at all, so they are an input rather than a later review.

3. Decisions this chapter supports#

• What must be instrumented before a long or expensive run is authorized at all.
• What minimum record one task execution has to leave behind, and where each of its fields comes from.
• How to split storage into separate planes, so that retention, privacy, and promotion rules attach to the right bucket instead of the loosest one.
• What production observability baseline to adopt at deployment — and, just as importantly, what that baseline does not give you.
• Whether the telemetry pipeline is itself trustworthy. Somebody has to check the checker.
• In what order a production failure re-enters the lab.
• Whether an automated or learned component still holds its licence to run unattended, or whether a de-automation trigger has fired.
• When the next periodic self-audit of your own methodology is due.

4. Normative principles#

5. Default procedure#

5.1 Land the telemetry floor before the run, not after#

The telemetry floor is the set of things that must already be recording before you start anything long or expensive. It is a floor rather than a wish list because of the asymmetry in §1: each row below costs almost nothing to write while the run is happening, and cannot be obtained at any price once it is over.

MUST: server/engine logs append across restarts. A log that truncates on restart silently destroys exactly the sessions a forensic reconstruction needs most (§9). SHOULD: at Tier 1, the floor MAY be trimmed to run/phase, per-invocation, and dual-clock rows only (§6); at Tier 2+ the full table is mandatory.

5.2 The minimum trajectory record#

A trajectory record is what you keep about one task execution so that somebody can still understand it later. The test it has to pass is whether a reader who was not there can replay the execution in their head from the record alone. The list is long because that test is demanding.

Every task execution MUST produce a record carrying at least these fields. This is the normative field set; a project MAY extend it, but MUST NOT ship fewer fields at Tier 2+ (§6).

5.3 Data-plane separation#

Never let one plane's access rule stand in for another's. A read permission on retrieval content says nothing at all about a read permission on trajectory PII.

5.4 Production observability baseline#

[ADAPT: NV-NIMOBSERVABILITY-001] (verified live 2026-08-21; page last updated 2026-08-19). Three mechanics here are worth following as they are written.

• A Prometheus-compatible metrics endpoint that exposes the serving engine's own metric vocabulary unmodified rather than a reinterpreted one. The verified pattern passes the backend's native metrics straight through instead of rewriting them.
• OpenTelemetry tracing keyed to a W3C traceparent header, so a single request can be followed across every hop it makes.
• Structured logs, one JSON object per line.

What is missing, and why this is ADAPT rather than FOLLOW: the source documents no experimentation minimum. It is a production-serving floor and nothing more. Supply the experimentation floor yourself, from §5.1 and §5.2, and do not assume production observability alone is sufficient before a long experimental run.

SHOULD: adopt one metric vocabulary across every serving tier — dev, shadow, canary, production. Two dashboards that each define "requests per second" their own way cannot be compared, and that is the same trap as comparing two performance-benchmark tools with different metric definitions (06).

[ADAPT: NV-ATIF-001] — if trajectories need to leave the local store, for cross-team sharing or a multi-tool pipeline, a standard interchange format is worth adopting over a bespoke export. What's missing: it is an interchange format across tools, not a canonical local schema. Keep §5.2's table as the schema of record, and use the interchange format only at the boundary where trajectories actually cross a tool or organizational boundary.

5.5 Telemetry-pipeline correctness#

Who audits the instruments' instruments. Five checks:

1. Cross-source consistency checks. Compare the same fact from two independent sources on every run — realtime against monotonic clock offset, client-measured against server-reported token counts, an events stream against a preserved log — and alert when they diverge past a pre-registered tolerance.
2. Synthetic-span injection. Periodically push a known synthetic task through the full pipeline and verify that every expected field arrives with the expected shape. A synthetic span that goes missing is the pipeline's own reachability ceiling (03), turned inward on itself.
3. Clock-skew probes. Sample the realtime/monotonic offset on a cadence that is independent of any experiment, so skew is caught before it contaminates a comparison rather than after (02).
4. Telemetry regression tests. Run these checks inside the same integrity-gate machinery that protects the eval suite — static integrity gates (03), full-pipeline rather than sampled, because they exist to catch exactly the defects a sampled check would miss.
5. Version telemetry-pipeline changes like a suite release. A schema or collection change gets a version, a changelog entry, and — at Tier 2+ — an explicit statement of which prior numbers it invalidates and which stay comparable.

5.6 Production failure harvesting#

Your production system is producing, for free and continuously, the one dataset your evaluation suite does not have: real failures, on real inputs, drawn from the actual population you serve. Every item in it is a case your current system demonstrably gets wrong, and it arrived without anyone having to imagine it. No other feedback loop available to you is worth as much per item.

Failure harvesting is what turns that stream into improvements. The instinct on seeing a pile of real failures is to train on them, and the entire discipline is in refusing to do that first.

1. Hold out a stream of annotated production examples specifically for regression testing — a named, mature pattern [EXT-OPS-002] — kept separate from the suites you use for capability claims (regression suite vs. capability suite, 03).
2. Apply the project's privacy filter (13) before anything from production crosses into the lab's stores. This MUST happen before harvesting, not as a later cleanup pass (13).
3. Triage every harvested failure against the canonical failure taxonomy (03). Is it already representable in the current suite (an eval gap, RC-1)? Is it a retrieval or context gap (RC-3)? Or is it a candidate for training data (RC-10) — which it becomes only once the ladder's cheaper rungs are evidenced exhausted (07, 09)?
4. Route in ladder order, per the principle above: eval-suite candidate first, then retrieval/context fix, only then training-data candidate. And a harvested record becomes a training-data candidate only after component-level provenance qualification (09 §5). A trajectory that embeds a managed-provider output — an escalation tier's response, a provider-generated label — is not training-admissible just because you own the record it sits in.
5. Feed routed items into the relevant chapter's intake — 03 for suite refresh, 09 for training data — with the trajectory record (§5.2) attached as provenance.

5.7 Drift and monitoring of automated/learned components#

This is where the third position from §1 is either renewed or revoked. A component running unattended is doing so on evidence that was collected once, and the four monitors below exist to notice when that evidence has quietly stopped describing reality.

Every threshold above is a consequence-bearing tolerance (04): pre-registered, with a named ABORT / RECALIBRATE / PROCEED-WITH-DECLARED-CEILING response attached before any data arrives. It is not a number on a dashboard that somebody is trusted to notice.

The graduation-reversal condition — de-automation — is the same tolerance read backwards. When it fires, the component reverts to observe-only automatically. Not when someone remembers to intervene.

Validation status: this playbook has no internal execution record of a component running long enough past graduation to exercise a drift trigger. What would validate this design: a graduated component instrumented per the table above, run past at least one pre-registered threshold (or one full recalibration cycle), with the response executed and recorded as evidence.

5.8 Periodic methodology audit#

[FOLLOW: EXT-OPS-002] — a rubric-based self-audit, scored as the minimum across categories rather than the average. This is a mature, external, consensus practice, and the scoring choice is the point of it: a system's overall test-and-monitoring maturity is only as good as its weakest category, and an average would let three strong categories conceal one absent one.

Run it annually or at every major release. At Tier 3, run it before every high-consequence promotion (10). Score categories SHOULD mirror this chapter's structure: telemetry-floor completeness, data-plane hygiene, production-observability coverage, telemetry-pipeline correctness, failure-harvesting throughput, and drift-monitoring coverage of automated components.

5.9 Closing the loop back to chapter 03#

The look ledger's refresh trigger (03, 04) and §5.6's harvesting output are the same loop seen from its two ends. Production evidence becomes suite-refresh input; the refreshed suite becomes new held-out evidence; the refreshed suite ships and starts generating new production evidence.

This is the second structural rule of the default lifecycle (00 §5): the loop closes back into evaluation, in ladder order, before it closes into training.

6. Project adaptation parameters#

• Resource-sampler interval — calibrate against run length and the granularity the autopsy will need. A run measured in minutes needs a tighter interval than one measured in days.
• Retention window per data plane — calibrate against the project's regulatory and residency constraints (01) and the clean-room boundary (13). Never default to "indefinite" without a stated reason.
• Telemetry-completeness threshold (§8) — calibrate against how much missing telemetry the downstream autopsy or decision can actually tolerate. State it as a consequence-bearing tolerance, not a vague aspiration.
• Drift-monitoring thresholds (§5.7) — calibrate against the component's blast radius and the stakes tier of the decisions it affects. A routing layer feeding a Tier-3 decision needs a tighter trigger than a Tier-1 exploratory one.
• Failure-harvesting sample rate (§5.6) — calibrate against production volume and the stratification needs of the suite it feeds (03).
• Methodology-audit cadence (§5.8) — calibrate against release cadence and stakes tier. Annual is the floor; per-major-release or per-high-consequence-promotion at Tier 3.

7. Decision gates and stopping conditions#

8. Metrics and formulas#

Telemetry completeness. How much of the floor is actually arriving.

Completeness = F_pop / F_req

where F_pop is the count of required fields actually populated in a sampled record (or run), and F_req is the count of fields the project's frozen telemetry-floor definition (§5.1, §5.2) requires. Expressed as a percentage.

Worked example (illustrative). A project defines a 20-field required floor. A spot-check of a smoke run finds 18 populated — two are silently null because of a harness defect. Completeness = 18/20 = 90%. The floor requires 100% on required fields before a run is trusted for its intended decision, so 90% triggers RECALIBRATE: fix the harness before authorizing the long run this floor exists to explain. That is the same ABORT / RECALIBRATE / PROCEED-WITH-DECLARED-CEILING vocabulary as any other consequence-bearing tolerance (04).

Drift statistics — reference only. No formula is prescribed here. Detecting population shift — comparing a monitored feature's production-time distribution against its measured-time distribution — is a standard statistical problem with several admissible standard tools, such as a distributional-distance statistic chosen to match the feature's type. status: doctrine — not yet exercised — this playbook has no internal execution record for any specific choice. Pick one appropriate to the feature type, and pre-register its threshold as a consequence-bearing tolerance (§5.7).

Clock-skew measurement and the MDE / effective-N machinery are defined in chapters 02 and 04 respectively. This chapter consumes them and does not redefine them.

9. Failure modes and anti-patterns#

• The dashboard as record of record (condition: whenever a trace-visualization or metrics UI is treated as the sole store of trajectories or evaluation results). A dashboard MAY mirror the record. It never replaces it (13).
• Success-gated trajectory storage (condition: whenever raw or failure-path output is persisted only after successful parsing). The population you most need to study is the failures, and that is exactly the population a success-gated store drops.
• Truncating logs across restarts (condition: whenever a long-running process's logs reset instead of append on restart). Forensic reconstruction depends on precisely the sessions this destroys first.
• Retrofitting telemetry during the postmortem (condition: always). Forensics cannot be reconstructed from data that was never captured. "We'll add it during the writeup" is a decision not to have the data.
• Activity telemetry substituting for outcome telemetry (condition: whenever token counts, tool-call counts, or self-reported completion stand in for a verified outcome). These are diagnostic signals and they are trivially inflated, so optimizing them directly is a textbook Goodhart failure. Keep activity and verified-outcome telemetry in separate columns, never blended into one score.
• Ground truth inline in the decision record (condition: whenever a production routing or gate log stores the outcome label at write time instead of joining it afterwards). This is exactly the leakage surface that ground-truth isolation (13) and the gold-label boundary (04) exist to prevent, reintroduced one layer down.
• Automation running past a fired drift trigger (condition: whenever a pre-registered monitoring threshold has fired and the component has not reverted to observe-only). Observe-only graduation (08) is conditional; treating it as a one-time achievement is the failure mode §5.7 exists to close.
• Undocumented internals scraped to fill a vendor gap (condition: whenever a vendor tool does not reliably expose a needed signal and the response is reverse-engineering rather than declaring it unavailable). An undocumented internal can change without notice and silently break the pipeline it was patched into. Record the gap instead (§5.4's adapter rule).

10. Vendor recipes#

The structural finding of this playbook (00 §10) shows up here in its sharpest form. Production observability is vendor-mature. The experimentation floor and the drift-monitoring layer are not, and this chapter supplies both.

[ADAPT: NV-NIMOBSERVABILITY-001] — production observability mechanics (§5.4): metrics endpoint, distributed tracing, structured logs. Verified live 2026-08-21 (page last updated 2026-08-19). ADAPT because the source states no experimentation minimum; the decision layer this playbook adds is exactly §5.1 and §5.2.

[REFERENCE: NV-DATAFLYWHEEL-001] — deprecated April 2026, with no successor named as of verification (2026-08-21). Do not adopt the artifact. The shape of the loop it demonstrated remains a usable design reference for §5.6 and §5.8: log production examples, stratify them, run a comparison across baseline and adapted candidates, score that comparison with a judge, and gate promotion on a human. Its own README described it as "flashlight, not autopilot", which is the right posture. The trap worth carrying away: the source's own deprecation notice was not consistently reflected in the vendor's other documentation, which kept cross-linking and promoting it after the notice went up. Verify a source's deprecation status at the source, rather than letting inbound links imply currency.

[FOLLOW: EXT-OPS-002] — the self-audit rubric (§5.8): mature, consensus, verified 2026-08-21. Min-across-categories scoring, and its production-failure-harvesting pattern underwrites §5.6.

[ADAPT: NV-ATIF-001] — trajectory interchange format (§5.4), current and active as of 2026-08-21. Adopt at interchange boundaries only. It is not a substitute for the canonical schema in §5.2.

11. Worked examples#

• SCENARIO-05 — an invented case in which telemetry preserved while a long evaluation round was running is what later lets a performance autopsy (06) reconstruct where the hours went and run the counterfactual that refutes an intuitive hardware purchase. None of that reconstruction is available unless §5.1's floor was already landed when the round happened.
• SCENARIO-12 — an invented case in which dual-clock stamps surface a systematic clock-skew artifact that had been silently biasing every published latency. The fix follows directly from having captured both clocks, not from anyone suspecting the skew in advance.
• Illustrative — the completeness calculation in §8 (a 20-field floor, 18 populated, 90% completeness, RECALIBRATE before the long run it exists to explain).

12. Outputs and artifacts#

• A landed telemetry floor (§5.1), verified via the completeness check (§8), before the next long or expensive run.
• An instantiated trajectory-record schema (§5.2) — the store or log format the project actually writes.
• A data-plane separation map (§5.3), feeding chapter 13's privacy classification.
• A deployed production observability baseline (§5.4) — metrics endpoint, tracing, structured logs.
• A versioned telemetry-pipeline correctness test suite (§5.5), run alongside the eval suite's integrity gates.
• A failure-harvesting backlog with ladder-order triage (§5.6), feeding 03 and 09.
• A drift-monitoring plan with pre-registered consequence-bearing tolerances (§5.7) for every graduated automated or learned component.
• A periodic methodology audit record (§5.8), feeding CHANGELOG.md and any method decision records it triggers.

13. Sources#

Gap dispositions in this chapter:

G5 (post-deployment drift monitoring of learned/routed components): COVERED-AS-DOCTRINE-NOT-YET-EXERCISED (§4, §5.7, §7). The monitoring design and its consequence-bearing thresholds are stated in full. No internal execution record exists for a component running past a fired trigger, and that is marked explicitly wherever it matters.

G12 (telemetry-pipeline correctness): COVERED (§5.5 procedure, §7 stop condition, §9 anti-pattern). The cross-source consistency checking and multi-source forensic reconstruction behind it come from one project's practice, whose records are not published here; [SCENARIO: SCENARIO-05] and [SCENARIO: SCENARIO-12] are invented illustrations of the same reasoning rather than the evidence for it.