--- name: enrich-from-pages-you-do-not-control description: Fill fields from pages somebody else serves, at volume, without recording a guess as a fact or a refusal as an answer. runbook: schema_version: v1 semver: 1.0.0 profile: P1 min_runtime_profile: R1 domain: data-pipelines targets: - linux trigger: manual execution: human-with-agent risk: reversible-write blast_radius: service duration: ">30m" capabilities: - cli:curl publisher: std lang: en license: CC-BY-4.0 inputs: sample_url: type: string required: true description: one page from the family being measured field: type: string required: false description: the field whose coverage is in question steps: - id: s1 kind: check title: Fetch one page, read its status code tool: cli:curl command: curl -sS -o /dev/null -w "%{http_code}" "{{inputs.sample_url}}" expect: a status code — 404 and 410 mean gone, everything else does not assert: stdout_matches: ^[1-5][0-9][0-9]$ on_fail: escalate:s10 next: s2 - id: s2 kind: decision title: What did the fetch actually get? question: A usable page, a refusal, an empty shell that needs a browser, or a genuine 404 or 410? branches: refused: s3 needs_browser: s4 gone: s9 default: s5 - id: s3 kind: action title: Record the refusal as a refusal risk: reversible-write expect: the record says it could not be checked, not that it is fine and not that it is dead on_fail: escalate:s10 next: s9 - id: s4 kind: action title: Queue it for a browser, after the cheap pass risk: read-only expect: it waits in a separate queue; the plain fetches finish first on_fail: escalate:s10 next: s9 - id: s5 kind: decision title: Does the page carry structured markup? question: Is there machine-readable markup for what you want, or only prose? branches: prose_only: s6 default: s7 - id: s6 kind: action title: Write a rule for this family, not for all risk: read-only expect: a rule scoped to this family of pages, measured on its own sample on_fail: escalate:s10 next: s7 - id: s7 kind: decision title: Did this field ever arrive? question: Across the sample, did the field come from the markup even once? branches: never: s8 default: s9 - id: s8 kind: action title: Stop promising the field, or take it from text risk: reversible-write expect: the field is either derived from prose and labelled as such, or it is not offered on_fail: escalate:s10 next: s9 - id: s9 kind: action title: Write what was read, and leave the rest empty risk: reversible-write requires_approval: true expect: no field carries a guess; an unknown stays unknown on_fail: escalate:s10 next: end:success - id: s10 kind: escalate title: Stop before the batch runs to: team:platform source: url: https://runbooks.directory/r/std/enrich-from-pages-you-do-not-control/source.md fetched_at: 2026-09-04 upstream_state: current --- ## When to reach for this Records are to be filled from pages on servers you do not run — listings, profiles, catalogues, registries — at a volume where nobody will read the results one at a time. Every number below is from one such measurement, on a sample small enough to read by hand. Yours will differ; the shapes will not. ## A refusal is not an answer, and wording is not a status Two mistakes, and both fill a database with confident wrong values. **Treating "blocked" as "fine".** Fetches that were refused — a rate limit, a bot check, a network that answered with a challenge — came to 43% of checks in one measurement. Recorded as successes, they mean the record was never actually verified, and publishing on that basis brought 8% dead listings back onto a public surface within twelve days. **Judging that a page is gone by what it says.** Only the status codes for *gone* mean gone: 404 and 410. A page that says "this is no longer available" in its body may be a template, a soft error, or a redirect to a search. Of 132 pages marked gone by their wording, 32 were alive. Read the code; the words are a different signal entirely. So the outcome of a fetch has at least four values, not two: usable, refused, empty, gone. A pipeline with only "ok" and "dead" has to put the other two somewhere, and wherever it puts them is wrong. ## Structured markup covers a third, and the rest need work per family Where pages publish machine-readable markup for what you want, one parser reads all of them. Measured: a third did. The remaining families served perfectly good HTML with no markup at all, and each needs its own rule. Plan for that. A single general parser that works on the first family looks finished and covers a third of the corpus, and the gap does not announce itself — it looks like missing data rather than a missing rule. ## Some pages are not readable without a browser, and a browser is twenty times the price Pages rendered entirely by script return a shell with none of the fields in it. In one measurement that was about 9% of links, and no amount of parsing helps: the content is not in what the server sent. Order matters and only one order is affordable: **the whole batch over plain HTTP first, then the empty and refused ones queued for a browser.** Browser-first costs roughly twenty times as much for the same result, and most of the corpus never needed it. ## A field that never arrives should not be offered One field in that measurement never came from markup — not once, across every family. It is therefore empty on the overwhelming majority of records, and a filter built on it silently hides almost everything. Two honest options: derive it from the prose and label it as derived, or do not offer the field. What is not honest is a field that is populated often enough to look real and rare enough to be useless — see `std/decide-where-a-model-is-needed`, which is the same lesson from the other end. ## A guess is worse than a blank Order the sources for each field from reliable to speculative, and stop before the speculation. An empty value in the output is honest and a reader can act on it: they know to look. Half a phrase, extracted because something had to be, is indistinguishable from a real value and will be read as one. ## Do not speed it up The instinct on a slow batch is more concurrency, and it is the one change that ends the batch. Measured against one large source: four workers hit a rate limit after about 1400 requests; two workers with a short pause between them completed 1500 without a single refusal. Slower finished sooner, because the fast run had to be restarted with the refusals cleaned up. Where a source publishes a limit, that limit is the design input; where it does not, find the number once and write it down, because the next person's instinct will also be to raise it.