Templates
Compute Demand LedgerTwo things can be true about the same month: the machine was pinned for a twenty-hour round, and the project has barely used any compute all year. From inside the week that round happened in, they feel like one fact. …Eval Suite Release ContractSooner or later somebody will set a score from this suite beside a score taken on a different version of it, and read the gap as the system getting better. It may just be the test that changed. This document is what s…Experiment ContractYou are about to compare two systems. Somewhere in the middle of the run you will be tempted to move a threshold by two points, drop the stratum that is behaving oddly, or read a small gap as a win. From the inside, n…Method Decision RecordSix months from now somebody who was not in the room will ask why the work is done this way. If nobody can answer, the question gets argued again from the start, by people with less evidence than you have in front of …Operational HandoffSomething is behaving oddly in production, and the person looking at it is not the person who built it. Maybe that is a new operator. Maybe it is you, four months later, with none of it still in your head. Either way,…Performance AutopsyA long run finished. You know the total it cost and nothing about where that total went — and the proposals arrive anyway: a bigger card, a different runtime, a second node. Every one of them is a guess about a bottle…Prediction LedgerBefore you run anything, you already expect something — otherwise you would not have chosen to run it. This ledger is where that expectation gets written down, one line per metric, while the answer is still unknown. W…Project ProfileThe intake questionnaire you fill in first: 21 questions about what you are building, what constrains it, and what it costs when the system is wrong. QUICKSTART routes on the answers, and the rigor dial reads them to …Test Look LedgerHeld-out evaluation data is a consumable. Every time somebody reads a held-out split's outcomes, some of it is used up — because from that moment on, decisions start being made in the light of what was seen there, and…
This index is generated from the template files; the files themselves are canonical.