Yash Verma / Cloud runbook

Pod in CrashLoopBackOff

A container keeps exiting and Kubernetes keeps restarting it with a growing delay. All playbooks.

See the restart count and the reason for the last exit. Look at Last State under each container, and at the events at the bottom.
Describe a pod

Shows a pod's status, container states, restart reasons and recent events.
```
kubectl describe pod <pod-name> -n <namespace>
```
pods, events, debugging
Read the logs of the run that crashed, not the one that is starting now.
Get logs from previously crashed pod

Useful for checking why a pod crashed before it restarted.
```
kubectl logs <pod-name> --previous
```
logs, crash, debug
If the reason is OOMKilled, compare the pod's memory with its limit, then raise the limit or fix the leak.
PromQL: Pod memory usage

Working-set memory per pod in MiB: the figure kubectl top shows and the kubelet uses for eviction.
```
sum by (pod) (container_memory_working_set_bytes{namespace="<namespace>", container!=""}) / 1024 / 1024
```
prometheus, promql, memory
If the exit code is 1 or 2 straight after start, it is usually configuration: a missing environment variable, secret or config file. Check what the pod actually received.
Get running pod YAML

Exports the live YAML configuration of a running pod.
```
kubectl get pod <pod-name> -o yaml
```
yaml, export
If the crashes began with a new release, roll back first and investigate after.
Rollback a deployment

Rolls back a deployment to the previous revision.
```
kubectl rollout undo deployment/<deployment-name>
```
rollback, undo, deployment