Writing a runbook

Audience: you have a procedure that works and you want it written down in a form an agent can be held to.

Everything here is practice. Where a rule is normative, it links to the section that makes it so — this guide never restates the specification in different words, because two statements of one rule eventually disagree and then nobody knows which is the rule.

Start with a check, not an action

A procedure that begins by changing something is not diagnosing, it is guessing (§18.6).

std/disk-pressure-triage is the whole guide in miniature. It could have started with a deletion — that is what somebody wanted at 3am — and instead its first step is df, its second is du, and it stops. Finding out what is filling a volume before deleting anything is what makes the deletion an informed act rather than a hopeful one.

One node, one action, one expected result

"Check the lag and restart the group" is two steps. Splitting them is not tidiness: a supervisor scopes permissions to the current step (§13.2), so a step that does two things holds the privileges for both at once, and the diagnostic half carries the destructive half's rights.

Titles are imperative and short

Up to 48 characters, a verb first. std/kafka-consumer-group-stuck-restart uses "Capture current lag", not "Lag problem" — a reader scanning a graph is looking for what happens, and a noun phrase makes them read the body to find out.

Every check needs an expectation and a route

expect says what should be true; on_fail says where the procedure goes when it is not (§6, invariant 4). Without the second, a supervisor reaching a failed check has to decide for itself, which is the improvisation the graph exists to prevent.

Write expect for the human. If the step will run under R2 supervision, add assert as well — a closed predicate over the observation, which is what a supervisor evaluates (§13.3). Prose is not evaluated, ever.

std/disk-pressure-triage shows why this matters more than it looks. Its du step asserts on the output rather than the exit code, because du exits non-zero on a shared mount whenever a subdirectory is unreadable. That is ordinary, not a failure — and an expectation written as prose reads perfectly and is wrong.

Destructive steps are always paired

Either an approval gate before, or a rollback after (§6, invariant 5). This is the rule the whole product exists to enforce, so it is structural: a record without it does not validate.

std/postgres-drop-stale-replication-slot does both. The drop requires approval, and its rollback_ref points at a step that recreates the slot. Note what the record says in its prose: recreating the slot does not recover the WAL already released. A rollback that does not fully undo the action should say so — the gate before the drop is where somebody decides that trade is acceptable, and they can only decide it if they know.

Loops are bounded or they are not loops

retry(n) is the only cycle a graph may contain, and n must be finite (§6, invariant 6). std/flaky-check-retry-budget exists to show the shape: without a limit, a condition that never settles becomes a loop that costs money until somebody notices.

Declare exactly the capabilities you use

capabilities[] is a privilege request, and a client checks it against its own allowlist before running anything (§12). Two failures follow from getting it wrong: a capability you use and did not declare is a blocking error, and one you declared and do not use is a warning, because over-declaration is how agents end up holding rights nothing in the procedure needs.

Inputs are declared, or the command is not runnable

A command interpolating {{inputs.group}} needs inputs.group declared with a type (§5). An unbound input does not make a command narrower — it makes it a different command, and --group "" against a production cluster is not a degraded version of --group checkout.

Mark anything that carries a credential secret: true, and it will be redacted wherever the run is shown to a human.

Twelve steps

Past twelve, a graph stops being readable at a glance, which is the only reason to draw one. Split it and link the parts.