Run reports
Normative for RUNBOOK.md §14. Schema: schema/run-report.json,
event binding: schema/run-report-envelope.json.
A run report is what a supervised run (§13.3, profile R2) leaves behind. It is an optional extension of this specification: the catalog does not depend on receiving any, anyone may collect them, and our collector is a reference rather than the one.
The rule the format exists to hold
Every field is either justified as actionable or excluded as a deanonymization risk. There is no third category, and "we will redact it before publishing" is not a design — it is a promise made by whoever has already received the secret.
So: no field accepts free text. Every string carries an enum, a pattern or a
format, additionalProperties is false at every level, and a report containing a
field this schema does not define is not a report. That constraint is mechanically
checked, not reviewed by eye (packages/schema/src/run-report.test.ts).
What an observer learns, field by field
The privacy review is this table. A new field is added only by extending it.
| Field | What it is for | What an observer learns about the reporter |
|---|---|---|
report_schema, schema_version, profile |
comparability across format versions | nothing |
runbook, version |
which procedure | that they run a published procedure; nothing about them |
content_hash |
what actually ran, past update drift | as above; a hash we have never seen says only that they run something we do not index |
outcome |
the headline fact | that one run of one procedure ended this way |
failed_step_id |
where it broke, so the aggregate is actionable | which step of a public document failed once |
failure_class |
what kind of breakage, from seven values | the category only — never the message, which is where secrets live |
executor_kind |
whether the procedure is fit for an agent | that they run procedures by hand, with an agent, or autonomously |
runtime_profile |
supervised and advisory failures are different facts | which supervision they deploy |
deviation_count |
that a graph does not match reality | how often their agent tried something the document forbids |
duration_bucket |
slow procedures are a real signal | one of four ranges. Never a precise duration: second-level precision plus submission time plus a rare runbook identifies a small organization |
env_class |
a failure in prod is not a failure in ci | one of four words. No cloud, no region, no cluster version: that is an infrastructure fingerprint |
reported_at |
ordering, aggregation windows | the day. Not the minute: a timestamp series is a movement log of one organization's incidents |
Deliberately absent, and to be kept absent: any error message, any command, any argument value, any hostname, any account or cluster identifier, any sender identity. A rotating anonymous token is needed for rate limiting and k-anonymity; it is not stored next to the report.
Publication rules
- Aggregates, not raw reports.
- k-anonymity: a counter is not published until it holds at least 5 reports from distinct tokens. Below that threshold the counter is withheld entirely — not rounded, not shown as "<5", since a suppressed cell next to published ones is itself a signal.
- An aggregate is republished as it changes; a report is never republished on its own.
Emission
Local by default and emitted nowhere. Sending a report anywhere is opt-in per run, by the operator (§13.3). A supervisor that phones home by default is not something anyone should install next to production, and a default that is easy to miss is a default that was chosen for the user.
Intake, and what it may keep
A collector — ours is a reference, not the one — validates each report against the schema and refuses a document with any field the schema does not define, whole rather than trimmed. A trimmed report is an unreviewed channel into a store whose aggregates are published.
What it keeps is counters and nothing else. The rotating token is used to decide whether this sender has already contributed to this counter, and then it is gone: it is not stored beside the report, and the mark left in the counter's contributor set is an HMAC under a salt belonging to that counter alone. The same sender therefore appears under unrelated marks in every counter they contribute to, and nothing in the store links two reports to one sender. That is the difference between correlation being impossible and being discouraged.
A per-token daily limit bounds how much one token can weigh in a counter. Two hundred reports a day, and the two hundred and first is refused with the reason.
What the threshold does and does not give you. It counts distinct tokens, and tokens are minted by whoever sends — so a party willing to mint five reaches it, and measuring this is a matter of running it rather than reasoning about it. The limit does not change that: bounding a token's volume cannot bound how many tokens exist. Nothing short of an identity can, and §14 refuses identity on purpose, so this is stated rather than implied.
What the threshold does give you is what it is for: no counter is published from a single ordinary reporter's runs, and no aggregate on this site describes one organization's afternoon. It is a floor against accident and casual observation, not against somebody deciding to defeat it. The sentence this replaces claimed the second, and the check it credited had never run.
A report about something we do not index
Accepted, and aggregated on its own content_hash. §14 binds a report to what actually
ran; requiring that hash to be one of ours would mean only our own catalog could report,
and the aggregate would describe our coverage rather than what people run.
Reports do not raise trust
T4 needs three distinct issuers, at least two of them not the publisher (§8). A report has no issuer — it has a rotating anonymous token, which is the point of the format — so no number of reports corroborates anything. The path from a run to a trust level is a signed run attestation, where somebody puts a key behind the claim.