Yash Verma / Cloud runbook

Node NotReady or needing replacement

A node stops reporting, or you need to take one out for maintenance without dropping traffic. All playbooks.

  1. List nodes with their status, version and IP addresses.

    kubectl get nodes -o wide
  2. Check the node's conditions (MemoryPressure, DiskPressure, PIDPressure, Ready) and recent events.

    kubectl describe node <node-name>
  3. Stop new pods landing on it.

    Cordon a node

    Marks a node unschedulable; running pods stay where they are.

    kubectl cordon <node-name>

    nodes, maintenance

  4. Move its pods elsewhere. PodDisruptionBudgets can make this wait; that is them doing their job.

    Safely drain a node

    Cordons the node and evicts its pods before maintenance, respecting PodDisruptionBudgets.

    Caution. --delete-emptydir-data deletes the data in emptyDir volumes on that node.

    kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

    node, maintenance, drain

  5. With Karpenter, delete the node's NodeClaim instead: Karpenter drains the node and provisions a replacement.

    kubectl delete nodeclaim <nodeclaim-name>
  6. Watch the replacement come up.

    Watch nodes being created and removed

    Streams NodeClaim changes as Karpenter provisions and consolidates nodes.

    kubectl get nodeclaims -w

    nodeclaims, watch, consolidation