decide-where-a-model-is-needed

Measure what rules can extract from a corpus before paying a model to read all of it, and tell a rule that is weak from a corpus that is silent.

Procedure

  1. start start. Goes next to s1
  2. check Draw a sample that favours nothing. Goes on_fail to s9; next to s2
  3. action Measure the rule on that sample, RO (read-only). Goes on_fail to s9; next to s3
  4. decision Does it fire often and answer wrongly?. Goes if mostly_wrong to s4; if default to s5
  5. action Add the context that tells them apart, RO (read-only). Goes on_fail to s9; retry to s2; next to s5
  6. decision Is the coverage low, and why?. Goes if absent_from_text to s6; if default to s7
  7. action Publish what is true instead of a filter, RW (reversible-write). Goes on_fail to s9; next to s7
  8. action Label fifty by hand, both directions, RO (read-only). Goes on_fail to s9; next to s8
  9. action Sort the corpus, then run the cheap pass, RW (reversible-write), requires approval. Goes on_fail to s9; next to __end_success
  10. escalate Stop before the full run. Terminal
  11. end success. Terminal

Open the graph on its own page — larger, printable, and an address you can send.

Walk it without running it — you answer what your environment would answer, and the path, the capabilities, the approvals and the distance back to an undo come out of the document. Nothing executes; there is no Run button here and there will not be one.

Trust

T2 · attested 2026-10-11 · runbooks.directory

Nobody has attested this record. That is the ordinary case: T2 is what the catalog can check on its own, and T3 needs somebody to run the procedure and sign for it.

  • Link liveness has not been checked: it needs a network and the catalog builds without one. Run pnpm gates --online where there is one.

Steps

  1. s1 check — Draw a sample that favours nothing
  2. s2 action — Measure the rule on that sample (read-only)
  3. s3 decision — Does it fire often and answer wrongly?
  4. s4 action — Add the context that tells them apart (read-only)
  5. s5 decision — Is the coverage low, and why?
  6. s6 action — Publish what is true instead of a filter (reversible-write)
  7. s7 action — Label fifty by hand, both directions (read-only)
  8. s8 action — Sort the corpus, then run the cheap pass (reversible-write) — requires approval
  9. s9 escalate — Stop before the full run

Capabilities

This is a privilege request. Check it against your own allowlist before running.

  • cli:psql — used by s1

Provenance

Source
https://runbooks.directory/r/std/decide-where-a-model-is-needed/source.md
Fetched
2026-09-04
Source last checked
not since it was captured on 2026-09-04 — recorded as current
Licence
CC-BY-4.0
How it got here
Written by this catalog's maintainers and reviewed in full.
Highest risk in this procedure
reversible-write

Take it with you

Run it under supervision

Runs here, on your machine, under supervision: every step is proposed before it happens, an approval gate stops for you, and a failed expectation goes where the procedure says rather than on to the next step. Nothing is sent anywhere. To supervise an agent that is doing the work instead, runbooks hook sits in its tool-call path.

npx @runbooks/cli run std/decide-where-a-model-is-needed@1.0.0 --as "$(whoami)" \
  --input corpus_table=<the table holding the documents to be read> \
  --input field=<the field being extracted, one at a time>