Observability commands

Check Elasticsearch health

Fetches the current health status of the ES cluster.

curl -X GET "localhost:9200/_cluster/health?pretty"

elasticsearch, elk, health

List Elasticsearch indices

Shows all indices in the ES cluster with their health, status, and size.

curl -X GET "localhost:9200/_cat/indices?v"

elasticsearch, elk, indices

Delete Elasticsearch index

Deletes one index.

Caution. Permanently deletes the index and its data.

curl -X DELETE "localhost:9200/<index-name>"

elasticsearch, elk, delete

PromQL: Pod CPU usage

CPU cores used per pod, averaged over 5 minutes. container!="" drops the pod-level series that would double-count.

sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="<namespace>", container!=""}[5m]))

prometheus, promql, cpu

PromQL: Pod memory usage

Working-set memory per pod in MiB: the figure kubectl top shows and the kubelet uses for eviction.

sum by (pod) (container_memory_working_set_bytes{namespace="<namespace>", container!=""}) / 1024 / 1024

prometheus, promql, memory

Restart the Grafana server

Restarts the grafana systemd service.

sudo systemctl restart grafana-server

grafana, systemd, restart

Query Prometheus API

Fetches current metric values from the Prometheus HTTP API.

curl -s "http://localhost:9090/api/v1/query?query=up" | jq .

prometheus, api, query

Check Prometheus targets

Lists all scrape targets and their health status.

curl -s "http://localhost:9090/api/v1/targets" | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

prometheus, targets, health

List Grafana datasources

Lists all configured data sources in Grafana via API.

curl -s -H "Authorization: Bearer <service-account-token>" http://localhost:3000/api/datasources | jq .

Grafana service account tokens replace the old API keys.

grafana, datasource, api

Check Alertmanager status

Shows the current status and active alerts in Alertmanager.

curl -s http://localhost:9093/api/v2/alerts | jq .

alertmanager, alerts, status

Check Node Exporter metrics

Fetches raw metrics from Node Exporter.

curl -s http://localhost:9100/metrics | grep node_cpu_seconds_total | head -5

node-exporter, metrics, cpu

Query Loki logs

Queries Loki for logs matching a label selector.

curl -G -s "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query={job="varlogs"}' | jq .

loki, logs, query

PromQL: HTTP error rate

Calculates the percentage of 5xx responses over the last 5 minutes.

sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100

promql, errors, sre

PromQL: P99 latency

Calculates the 99th percentile request latency from a histogram.

histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

promql, latency, p99

Check Metrics Server

Verifies that the Kubernetes metrics-server is running and healthy.

kubectl get deployment metrics-server -n kube-system

metrics-server, k8s, health

Blackbox Exporter probe

Tests HTTP endpoint availability using Blackbox Exporter.

curl -s "http://localhost:9115/probe?target=https://example.com&module=http_2xx" | grep probe_success

blackbox, probe, uptime

Query Jaeger traces

Fetches recent traces from Jaeger for a specific service.

curl -s "http://localhost:16686/api/traces?service=my-service&limit=5" | jq '.data[].traceID'

jaeger, tracing, distributed

cAdvisor container stats

Fetches container-level resource stats from cAdvisor.

curl -s http://localhost:8080/api/v1.3/containers | jq '.spec.cpu'

cadvisor, containers, metrics

Check Prometheus rule files

Validates the syntax of alerting and recording rules before you load them.

promtool check rules rules.yml

prometheus, rules, validate

Check an Alertmanager config

Validates alertmanager.yml and summarises its receivers and routes.

amtool check-config alertmanager.yml

alertmanager, validate

Silence an alert

Silences an alert by name for two hours, with a comment saying why.

amtool silence add alertname="<alert-name>" --duration=2h --comment="<reason>" --alertmanager.url=http://localhost:9093

alertmanager, silence

Reload the Prometheus configuration

Tells Prometheus to re-read its configuration and rule files without a restart.

curl -X POST http://localhost:9090/-/reload

Works only when Prometheus runs with --web.enable-lifecycle.

prometheus, reload