--- name: decide-where-a-model-is-needed description: Measure what rules can extract from a corpus before paying a model to read all of it, and tell a rule that is weak from a corpus that is silent. runbook: schema_version: v1 semver: 1.0.0 profile: P1 min_runtime_profile: R1 domain: llm-ops targets: - postgres trigger: manual execution: human-with-agent risk: reversible-write blast_radius: service duration: ">30m" capabilities: - cli:psql publisher: std lang: en license: CC-BY-4.0 inputs: corpus_table: type: string required: true description: the table holding the documents to be read field: type: string required: true description: the field being extracted, one at a time steps: - id: s1 kind: check title: Draw a sample that favours nothing tool: cli:psql command: psql -Atc "select count(*) from {{inputs.corpus_table}} where substr(md5(id::text), 1, 2) < '1a'" expect: a sample of roughly a tenth, chosen by hash rather than by date or source assert: exit_code: 0 on_fail: escalate:s9 next: s2 - id: s2 kind: action title: Measure the rule on that sample risk: read-only expect: how often it fires, and how often what it found was right on_fail: escalate:s9 next: s3 - id: s3 kind: decision title: Does it fire often and answer wrongly? question: Of the matches you read by hand, how many were the thing you were looking for? branches: mostly_wrong: s4 default: s5 - id: s4 kind: action title: Add the context that tells them apart risk: read-only expect: the rule requires a neighbouring word that means this, and forbids ones that mean something else retry: max: 1 target: s2 on_fail: escalate:s9 next: s5 - id: s5 kind: decision title: Is the coverage low, and why? question: Does the rule miss what is there, or is the answer absent from most documents? branches: absent_from_text: s6 default: s7 - id: s6 kind: action title: Publish what is true instead of a filter risk: reversible-write expect: a flag saying whether the document states it at all, rather than a field pretending to be complete on_fail: escalate:s9 next: s7 - id: s7 kind: action title: Label fifty by hand, both directions risk: read-only expect: what the rule resolved is right, and what it called absent is genuinely absent on_fail: escalate:s9 next: s8 - id: s8 kind: action title: Sort the corpus, then run the cheap pass risk: reversible-write requires_approval: true expect: documents rules can answer are not sent to a model at all, and the rest are asked only about what is missing on_fail: escalate:s9 next: end:success - id: s9 kind: escalate title: Stop before the full run to: team:platform source: url: https://runbooks.directory/r/std/decide-where-a-model-is-needed/source.md fetched_at: 2026-09-04 upstream_state: current --- ## When to reach for this There is a pile of documents — listings, tickets, reviews, filings — and structured fields are wanted out of them. A model will read all of it if you let it, and the bill scales with the pile. Before that, it is worth knowing which fields rules already answer, which ones rules answer *wrongly*, and which are not in the documents at all. Measured once on a real sample of one corpus, the split came out: rules sufficed for 34.9% of documents, a model was needed for one or two fields on 60.1%, and for everything on 5%. Sorting on that turned a full run of about $37 into about $16 — a third never sent, and most of the rest asked a short question instead of the whole schema. The numbers are not the point; the shape is, and you get your own by doing this. ## Sample by hash, not by convenience The first thousand rows are the oldest, or the newest, or one platform's. Take them by a hash of the identifier — `substr(md5(id), 1, 2) < '1a'` and its kin — and the sample favours no source, no company and no date. It costs the same and it is the difference between a measurement and an anecdote. ## A rule that fires often is not a rule that is right This is the trap, and it is worth stating in full. "A currency symbol near a number" fired on 29.8% of documents. Of twelve matches read by hand, half were not pay at all: a company's valuation, a funding round, a year's turnover, monthly food vouchers, a referral bonus. Coverage looked excellent and the field would have been wrong on half the corpus — silently, because nothing downstream re-reads a number that has already been extracted. What separated them was context, not the pattern. Require a word about *pay* within a short window — salary, compensation, gross, per year, per hour, and the same words in the languages the corpus is in — and forbid words about *company* money: valuation, revenue, funding, series, turnover, referral, voucher. After that, every match read by hand was real. So the measurement that matters is not how often a rule fires. It is how often what it found was the thing you wanted, and the only way to know is to read some by hand. ## Low coverage is sometimes a fact about the world Corrected, the same rule extracted a clean range from 1.5% of documents. Another 12.8% mentioned pay without a range — "up to", "starting from", in words. And 83.7% said nothing about it at all. The temptation there is to reach for a model, and it would not help: **there is nothing in the text to extract.** A model asked to find a salary in a document that does not state one will either say so, at cost, or invent one, at a worse cost. What is useful instead is the honest derived fact — *does this document state it at all* — which splits the corpus into the sixth that talks about money and the rest. That is a filter somebody can use. A salary field populated for one document in sixty-six is a filter that quietly hides everything. ## One refinement, then the question changes The loop between measuring and adding context is bounded at one pass on purpose. If a rule is still mostly wrong after the obvious context has been added, the next honest move is not a third pattern — it is to accept that this field needs something that reads, and to sort the corpus accordingly. Rules that take five rounds of tightening are rules that will break on the next corpus. ## Two passes, because two economies Bulk fields and per-item depth are different problems and mixing them is what makes a run expensive. **The bulk pass** covers everything and populates the fields people search and filter by. It runs once and re-runs when the rules change. Sorting decides its cost. **The on-demand pass** covers the few documents somebody is actually going to act on — dozens, occasionally hundreds. Verbatim requirements, obligations, the specific wording: rules cannot touch these, because there is no pattern, only prose. At that volume a stronger model and a longer prompt cost pennies. Splitting them removes the standing temptation to run everything through the expensive path for the sake of fields that matter on one document in a hundred. ## Fifty by hand before ten thousand by machine Every percentage above is measured; the *accuracy* of each rule was checked on about a dozen examples. Before a full run, label fifty by hand and check both directions: that what the rule resolved is right, and that what it called absent is not something it missed. It is half an hour. A wrong rule multiplied by ten thousand records costs more than that to find and much more to undo, because by then the wrong values look like data.