---
name: k8s-node-not-ready-drain
description: Drain a node stuck NotReady and reschedule its pods, after establishing whether the kubelet is gone or merely unreachable.
runbook:
  schema_version: v1
  semver: 1.0.0
  profile: P1
  min_runtime_profile: R2
  domain: cloud-infra
  targets:
    - k8s
  trigger: alert
  execution: human-with-agent
  risk: destructive
  blast_radius: cluster
  duration: 5-30m
  capabilities:
    - cli:kubectl
  publisher: std
  lang: en
  license: CC-BY-4.0
  inputs:
    node:
      type: string
      required: true
      description: node name
  steps:
    - id: s1
      kind: check
      title: Confirm the node is NotReady
      tool: cli:kubectl
      command: kubectl get node {{inputs.node}} -o "jsonpath={.status.conditions[?(@.type=='Ready')].status}"
      expect: the Ready condition is False
      assert:
        exit_code: 0
        stdout_matches: ^False$
      on_fail: escalate:s9
      next: s2
    - id: s2
      kind: decision
      title: Is the kubelet reachable?
      question: Does the kubelet answer, or has the node gone entirely?
      branches:
        unreachable: s3
        responding: s6
        default: s9
    - id: s3
      kind: action
      title: Cordon the node
      risk: reversible-write
      tool: cli:kubectl
      command: kubectl cordon {{inputs.node}}
      next: s4
    - id: s4
      kind: action
      title: Drain the node
      risk: destructive
      requires_approval: true
      rollback_ref: s5
      tool: cli:kubectl
      command: kubectl drain {{inputs.node}} --ignore-daemonsets --delete-emptydir-data
      next: end:success
    - id: s5
      kind: action
      title: Uncordon the node
      risk: reversible-write
      tool: cli:kubectl
      command: kubectl uncordon {{inputs.node}}
      next: end:aborted
    - id: s6
      kind: wait
      title: Wait for the node to recover
      until: the Ready condition returns to True
      timeout: 5m
      retry:
        max: 3
        target: s6
      next: end:success
    - id: s9
      kind: escalate
      title: Escalate to the cluster owner
      to: team:platform
  source:
    url: https://runbooks.directory/r/std/k8s-node-not-ready-drain/source.md
    fetched_at: 2026-09-03
    upstream_state: current
---

## When to reach for this

A node has been NotReady long enough to matter and its pods are not being rescheduled.
The decision that shapes everything after it is whether the kubelet is gone or merely
unreachable: draining a node whose kubelet is about to come back evicts pods that had
nowhere better to go.

## Notes

The drain is destructive and its rollback uncordons the node — which restores
scheduling, but does not bring back the pods that were already evicted. The approval gate
before the drain is where somebody decides that trade is acceptable, and they can only
decide it if they know that is the trade.