--- name: one-worker-for-a-single-session-credential description: Run an integration whose credential is one account session without losing that session — a second worker on the same key reads as a hijack, and the key does not come back. runbook: schema_version: v1 semver: 1.0.0 profile: P1 min_runtime_profile: R2 domain: data-pipelines targets: - aws trigger: manual execution: human-with-agent risk: irreversible blast_radius: tenant duration: ">30m" capabilities: - cli:pgrep publisher: std lang: en license: CC-BY-4.0 inputs: worker: type: string required: true description: the process name of the worker that holds the session queue_depth: type: string required: false description: how much is waiting, for judging whether a run finished or stopped steps: - id: s1 kind: decision title: Is the scheduled worker stopped? question: Is the cloud worker disabled, with no invocation still in flight? branches: still_running: s10 default: s2 - id: s2 kind: check title: No other copy is running on this machine tool: cli:pgrep command: pgrep -c -f {{inputs.worker}} expect: nothing — pgrep exits non-zero when it finds no match assert: exit_code: 1 on_fail: s3 next: s4 - id: s3 kind: decision title: Whose process is it? question: Is the process you found this run, an abandoned one, or somebody else's? branches: abandoned: s4 default: s10 - id: s4 kind: decision title: Is concurrency pinned to one everywhere? question: Do the worker's reserved concurrency, the queue's batch size and the deploy-time check all say one? branches: not_pinned: s10 default: s5 - id: s5 kind: action title: Start exactly one worker risk: reversible-write expect: one process, holding one session on_fail: escalate:s10 next: s6 - id: s6 kind: decision title: Did a duplicate-authorisation error appear? question: Has the provider reported this key as used from more than one place? branches: duplicate: s7 default: s9 - id: s7 kind: decision title: Is the session still alive? question: Does the provider still accept this key, or has it been revoked? branches: revoked: s8 default: s10 - id: s8 kind: action title: The owner reissues the key by hand risk: irreversible requires_approval: true expect: a new session key, issued by the account's owner, because nobody else can on_fail: escalate:s10 next: s9 - id: s9 kind: decision title: Did the run finish on its own? question: Did the worker end because the work was done, rather than stopping partway? branches: stopped_early: s10 default: s11 - id: s11 kind: action title: Write down what could not be reached risk: read-only expect: anything the run could not fetch is recorded, rather than read later as absent next: end:success - id: s10 kind: escalate title: Stop, before the key is spent to: team:platform source: url: https://runbooks.directory/r/std/one-worker-for-a-single-session-credential/source.md fetched_at: 2026-09-04 upstream_state: current --- ## When to reach for this An integration authenticates as **an account**, not as an application: a messaging platform, a scraper holding a logged-in session, a trading or ledger API keyed to one identity. The credential is a session, the provider counts limits per account rather than per connection, and it treats a second simultaneous client as somebody stealing the first. Reach for this before running one by hand, before raising its concurrency, and immediately if you see a duplicate-authorisation error. ## The rule, and what it costs to break **One worker. Always. Not for throughput — for the key.** Two clients on one session key are, to the provider, an account being used from two places at once, and the response is to revoke the key. Permanently. Not rate-limited, not throttled: gone, reissuable only by the human who owns the account, by hand, in whatever interface the provider gives people rather than programs. Nothing in the deployment can do it, which is exactly why nothing in the deployment can undo the mistake. This is not a caution written from the documentation. Raised to four workers, four started in the same second, produced 88 duplicate-authorisation errors between them, and the session died. While it was being reissued, 314 of 934 items were still reachable — the rest were not fetchable by anybody until a person had logged in again. So the limit is set in every place that can override it: the worker's own concurrency, the queue's batch size, and a check at deploy time that refuses any other value. Three places for one number looks redundant until you notice that each of the three can raise it alone, and that the cost of it being raised is not recoverable. ## The gap the tooling does not close Automated guards usually check the *scheduled* worker: is the cloud one disabled before a manual run. That is the check that is easy to write, and it is half of the danger. The other half is a second **local** process — another terminal, an abandoned run, a colleague on the same machine. Nothing checks for it, because the thing that would check is the thing being started twice. Two local runs collide first on database locks, which looks like a confusing failure, and then on the session, which is the failure that costs something. Look before starting. It is one command and it is the only guard there is. ## Spare keys are spares, not parallelism A configuration that lists several session keys invites the reading that the work can be divided between them. It cannot: the client takes the first live key and stops, and the others exist for the case where the first has died. Which also means the work is scoped to what *that* account can see. A second account's view is a second run, later, not a wider run now — and if nobody has ever done that run, the data from the other accounts is not missing, it was never attempted. Say which, because "we have everything" and "we fetched what one account could see" are different claims. ## Idle connections die during long work, and the write is what fails Fetching one large item can take minutes, and the database connection sits idle throughout. Managed databases close idle connections, so the failure arrives at the moment the work is written — after it was fetched. On one run over fifteen items, fourteen failed this way and the data they had already retrieved was lost with them. Reconnect and retry **exactly once**. An idle drop is not a data error, so retrying it is right; a second drop in a row is something else, and it has to reach the top rather than spin. One retry turned that run into zero failures; a loop would have turned it into a worker that never admits it is stuck. ## A duplicate error with a live key is not a reason to restart If the key still works after a duplicate-authorisation error, the reading is not "it recovered". It is that **something else is holding this session right now** — an abandoned process, a colleague, a scheduled worker somebody re-enabled. Starting again is starting a second client, which is the thing that spends the key. So that branch escalates rather than retrying. The only safe next move is to find what else is running, and there is no version of that a restart performs. ## Why most of this is decisions rather than checks A `check` is something a supervisor evaluates and routes on; a `decision` is a question a person answers. Nearly every step here is the second kind, and saying so is not bookkeeping. "Is the scheduled worker stopped", "is concurrency pinned to one in all three places", "did the run end because it was done" — none of these has a command that answers it in every environment, and a record that dressed them as checks would be declaring a supervision nobody can perform. One step here is genuinely checkable: whether another copy is running on this machine, which is one command with a defined exit code, and it carries an assert. The record still declares R2, because its risk is irreversible and R2 is what that implies. What R2 gets here is a supervisor that halts at each decision and asks — which is exactly what a procedure this expensive should do. ## What an agent may do here `execution: human-with-agent`, `min_runtime_profile: R2`, and the reissue carries `requires_approval` because it cannot be done by an agent at all — only the account's owner can log in as themselves. Every other step is safe to delegate to something supervised, which is the point of writing the dangerous one down separately.