Devops Incident Management

Orientation

DevOps Incident Management is the disciplined workflow a team executes when a delivered service deviates from its expected behavior in production. It matters the moment an alert fires, a user reports incorrect results, a health check turns red, or a metric breaches a defined threshold. This page covers the day-to-day tactics for moving from detection to resolution and back, using tools and patterns that exist in most engineering organizations. It does not prescribe a single platform’s UI or workflow; instead it names the concrete commands, flags, and data‑flow shapes that recur across Kubernetes‑based, container‑orchestrated, and bare‑metal environments. If your stack differs, the same logical steps apply—only the specific binary or API call changes.

Identify & Triage

The first minute of an incident determines whether the response is contained quickly or escalates unnecessarily. The reader’s immediate goal is to confirm the symptom, scope the impact, and decide whether this is a surface‑level glitch or a systemic outage.

Confirm the symptom with live data

• kubectl get pods -A --field-selector=status.phase=Running lists all running pods across namespaces. If a critical deployment shows CrashLoopBackOff, the failure is at the container runtime level.
• systemctl status myservice on a bare‑metal or VM host reveals whether the service manager considers the process active, dead, or failed.
• curl -sSf https://status.example.com/health returns 200 OK or an HTTP error. A 503 Service Unavailable alongside a downstream database error points to a dependency failure.

Read the immediate error text

• A pod spec may surface Error: container has run out of memory in kubectl describe pod <pod>. This is a hard limit hit, not a transient spike.
• journalctl -u myservice -n 50 --no-pager may show Failed to bind socket: Address already in use. The port is occupied by a stale process or a second instance started by a race condition in the deployment script.
• psql -c "SELECT error_string, count(*) FROM logs WHERE timestamp > now() - interval '5 minutes' GROUP BY error_string;" surfaces repeated constraint violations or connection timeouts.

Scope the impact

• kubectl get endpoints <service> -o yaml shows which backends are registered. If only one of three replicas responds, the incident is partial.
• docker ps --filter "status=running" on a host confirms whether the expected container is actually running or has been OOM‑killed without a restart policy.
• git log --oneline -5 on the deployment manifest reveals if a recent commit changed resource requests, image tags, or environment variables—common triage starting points.

Sharp edges

• Alert fatigue: many teams receive the same “high CPU” alert every hour because the metric threshold is too low. The sharp edge is that the first person to acknowledge the alert may silently dismiss it, assuming it’s a repeat, and by the time a human investigates, the service has already been degraded for twenty minutes.
• Misrouted pages: in some organizations, on‑call rotation is managed by a ticketing system’s automatic escalation. If the escalation policy is misconfigured, the page goes to an engineer who owns the dashboard but not the underlying service, leading to “not my problem” responses. Verifying the routing rule in the alertmanager or PagerDuty config before assuming it’s correct saves time.
• Partial truth from kubectl: the command returns the API server’s view, which may be stale if the kubelet hasn’t reported a node’s status yet. Always cross‑check with kubectl get nodes -o wide and ssh <node> when the API seems inconsistent.

Contain & Mitigate

Once the symptom is confirmed, the reader’s goal is to prevent further blast radius while the investigation runs. Containment is not always a full rollback; sometimes it’s a targeted action that stops the bleeding without reverting good changes.

Targeted container-level actions

• kubectl rollout undo deployment/<deployment> reverts the pod template to the previous successful revision. This is the fastest way to restore a known‑good state when the new image has a buggy migration script.
• kubectl scale deployment <deployment> --replicas=0 followed immediately by kubectl scale deployment <deployment> --replicas=3 forces a clean restart of all pods. Use this when a configuration change (e.g., an env var) is suspected but you want to keep the new image.
• docker stop <container_id> on a host where a runaway process is consuming I/O. Follow with docker restart <container_id> if the service has a systemd unit that manages its lifecycle.

Host‑level containment

• systemctl isolate multi-user.target on a single‑purpose server can kill all user sessions and free resources for diagnostic commands. Use only on nodes where you have out‑of‑band access.
• iptables -A INPUT -s <bad-ip> -j DROP or kubectl patch node <node> -p '{"spec": {"podCIDR": "<new-cidr>"}}' to isolate a node that’s flapping or compromised. Note that changing a node’s CIDR requires a cluster restart in many CNI plugins.

Feature‑flag toggles (if your platform supports them)

• Many services expose a /set-feature-flag endpoint that accepts {"dark_mode": false}. If the incident is caused by a newly rolled‑out flag, a single HTTP POST can disable it without a new deployment.
• When no flag exists, helm upgrade --set featureFlag=false . can patch a Helm chart’s values if the flag is configured via values.yaml.

Sharp edges

• kubectl rollout undo only works if the revision history is retained. If your cluster has historyLimit set low, or if you’re using a CI system that pushes fresh manifests without preserving old replicasets, the command may report “no revisions to undo.” Always verify kubectl rollout history deployment/<deployment> first.
• Scaling to zero and back up can trigger pre‑start hooks that fail if the underlying dependency (e.g., a database) isn’t yet reachable. The sharp edge is a pod that enters ContainerCreating then Error because it can’t resolve the DB hostname. Add a readiness probe with an initial delay that accounts for dependency startup time.
• Feature‑flag endpoints are often undocumented. Reverse‑engineering them from the front‑end code or OpenAPI spec is common, but the flag name may change between releases without a schema change, causing the POST to return 404 Not Found or 400 Bad Request silently. Keep a local map of flag names, default values, and the commits that introduced them.

Investigate & Root Cause

With the immediate blast radius contained, the reader now digs into why the incident happened. This phase relies on logs, metrics, and a systematic correlation of events.

Log aggregation and correlation

• kubectl logs -p <pod> prints the last container’s output, including the exit code and any panic stack traces. The -p flag prints the previous container’s logs if the pod has restarted, which is essential for “mystery restarts.”
• journalctl -u myservice --since "5 minutes ago" on a host shows the systemd service lifecycle, cgroup OOM kills, and any kernel messages (dmesg output interleaved) that explain why the process died.
• psql -c "SELECT * FROM pg_stat_activity WHERE state = 'active' AND query_start < now() - interval '10 seconds';" identifies long‑running queries that may be blocking other transactions. This pattern surfaces when a sudden traffic spike overwhelms connection pool limits.

Metrics and query patterns

• Prometheus queries that most teams reuse: rate(http_requests_total[1m]) to see if request volume spiked exactly when the incident started; container_memory_working_set_bytes{image="myimage:latest"} to confirm a memory leak; up{kubernetes_namespace!="kube-system"} to verify all expected services are up.
• grafana-cli eval "query1 > 2 * query2" in some deployments automates the “is this a spike or a sustained level?” check, but be aware that Grafana’s expression language varies by backend; the underlying Prometheus query shape remains the same.

Correlating across systems

• If the incident involves a database outage, pg_dump -Fc <db> | gzip > backup.sql.gz taken mid‑incident preserves the state for later forensic analysis. Do not rely on WAL archiving alone if the incident lasted long enough to rotate logs.
• etcdctl snapshot save /var/lib/etcd/snapshot.db on a Kubernetes control plane node captures the cluster state. This is only needed if you suspect etcd corruption causing pod scheduling failures.

Sharp edges

• Log truncation: many container runtimes default to a maxlogsize of 10MiB. If a service outputs verbose debug logs during a hot path, the log file rotates and the earliest entries are lost before you run kubectl logs. Always set --log-opts max-size=100MiB --log-opts max-file=5 in your pod spec if the incident may generate substantial output.
• kubectl logs -p only works if the pod has restarted at least once. If the pod has been running continuously for days and the incident is a slow degradation, you’ll see only the current session’s output. In that case, forward logs to an external collector ( Loki, Elastic, or a syslog gateway) before the incident, and query from there.
• Prometheus rate() and increase() functions behave differently on reset counters. If your service resets its internal request counter at midnight, rate(http_requests_total[1m]) may show a false drop. Use increase(http_requests_total[1m]) for absolute counts over the window, or align your queries to the service’s reset cadence.
• Database connection pool exhaustion often masquerades as “high latency” in application logs. The sharp edge is that pg_stat_activity may show state = 'active' but wait_event_type = 'Lock' and wait_event = 'relation'—the query isn’t blocked by the database; it’s waiting for a row-level lock caused by a missing index or a transaction that ran too long. Always check locked_relations and recent VACUUM status before assuming the DB is down.

Restore & Verify

Containment and investigation are complete. The reader now restores the service to a healthy state and confirms it’s serving traffic correctly.

Rolling back or re‑deploying

• helm rollback <release> <revision> restores the previous Helm release revision. This is equivalent to kubectl rollout undo but operates at the Helm level, preserving the values.yaml state of the undone revision.
• git revert --no-edit <commit-hash> creates a new commit that undoes the changes of the bad deployment commit. Push it to the repo; your CI system will automatically apply the new manifest to the cluster.
• docker compose up -d on a host running docker‑compose‑v2 rebuilds and starts services defined in docker-compose.yml. Use --build to force a new image build if the tag hasn’t changed but the Dockerfile was modified.

Post‑restart verification

• curl -sSk https://api.example.com/health | jq .status returns "passing" or "degraded". Pipe the exit code into a script that blocks further automation until the health check passes.
• kubectl rollout status deployment/<deployment> --timeout=300s waits for all new pods to reach Ready status. If the timeout fires, kubectl get pods -w in another terminal shows which pod is stuck and its current status.
• systemctl is-active myservice && echo "service up" is a quick shell check for a non‑containerized service. Combine with systemctl journalctl -u myservice -n 20 --no-pager to see the last log lines after restart.

Data integrity checks

• pg_dump -Fc <db> > post-incident-backup.sql.gz after the service is restored ensures you have a clean snapshot of the database state. Compare row counts or checksum against the pre‑incident backup if you suspect data loss.
• redis-cli INFO keyspace shows the number of keys and memory usage. A sudden drop in key count may indicate a cache purge or eviction policy that ran during the incident.

Sharp edges

• helm rollback only works if the Helm release was installed with helm install --generate-name and the Tiller (or newer Helm 3 release history) is intact. In Helm 3, revisions are stored in the Kubernetes API, but if you’ve deleted the release and re‑installed it, the rollback target is gone and you’ll get “no revisions to rollback to.” Always keep a Git repo of released Helm charts as the source of truth.
• git revert may conflict if the deployment manifest has been edited manually after the bad commit. The sharp edge is a merge conflict in k8s/deployment.yaml that requires manual resolution before the revert can complete. Have a standard manifest template that CI applies, minimizing manual edits.
• Health‑check endpoints often return 200 OK based on a simple SELECT 1 or echo ok, but they may not reflect real user‑facing functionality. A service can pass its internal health check while its downstream database is still recovering. Always chain at least two checks: an internal liveness probe and an external synthetic transaction (e.g., curl through the public API) before declaring the incident resolved.
• Docker‑compose up -d recreates containers but does not restart the host’s systemd services that may depend on them. If your stack includes a systemd unit that starts Docker Compose on boot (ExecStart=/usr/local/bin/docker-compose up -d), the unit may still think the service is down because compose’s exit code is 0 but the containers took longer than the unit’s timeout to become healthy. Adjust the TimeoutStartSec in the unit file or add a ConditionPathExists check.

Review & Improve

The incident is resolved. The final step is to ensure the same failure mode doesn’t re‑appear, and that the team’s incident process itself has learned something.

Post‑mortem artifacts

• A concise timeline: a list of timestamps (in UTC) for alert fire, first response, containment, root‑cause identification, resolution, and post‑mortem start. Most teams use a shared Google Sheet or a dedicated post‑mortem tool (e.g., Jeli, Incident.io) but the structure is always the same: when, what, who.
• A root‑cause statement: one sentence that avoids “human error.” Example: “The pod crashed because the node’s kubelet exceeded its --maximum-deadzone threshold due to a kernel upgrade that changed cgroup CPU quotas.” This pattern—technical cause, not blame—is the sharp edge that separates mature incident cultures from punitive ones.
• Action items: each must be specific, owned, and dated. “Add a readiness probe with initialDelaySeconds: 60 to the payment service deployment” is actionable. “Improve monitoring” is not; it must be broken into “Add Prometheus alert on container_memory_working_set_bytes exceeding 80% of request for payment-service.”

Process improvements

• Review the alert configuration. If the alert fired on a metric that also fires during normal deployments, add a for: 5m duration or a rate() expression to suppress noise. The sharp edge is that many teams keep the for duration at 0s because “we want to catch everything,” resulting in the alert fatigue described earlier.
• Update the runbook. If the incident revealed a gap—e.g., “we didn’t know which node had the problematic pod until we SSH’d into three hosts”—add a step to the runbook: kubectl get pods -o wide --selector=app=<app> before any manual investigation.
• Conduct a “pre‑mortem” on the next deployment. Walk through the deployment checklist and ask “what if the image has a missing migration?” or “what if the DB schema change fails halfway?” Document the mitigation for each “what if” so the next incident’s containment is faster.

Sharp edges

• Action items often become “shelfware.” The sharp edge is that a team may close the post‑mortem with five action items, none of which are assigned or tracked in a ticket system. Without a designated owner and a sprint‑planning slot, the item decays and the next incident repeats the same failure. Assign ownership in the same tool where the post‑mortem is recorded.
• Timeline accuracy depends on clock synchronization. If your nodes run NTP but one control‑plane node drifted 12 minutes during a kernel upgrade, the incident timeline you recorded will be slightly off. When in doubt, cross‑reference the Kubernetes API server’s metadata.creationTimestamp with the kubectl get events output, which timestamps against the API server clock.
• Post‑mortem templates vary widely. Some organizations require a “ blameless” section, others a “technical summary” first. The pattern that survives across teams is: timeline → impact → root cause → action items → verification. Stick to that order, and any organization can map it to their preferred format.

*This reference page is intentionally platform‑agnostic. The commands, flags, and data‑flow patterns shown exist in real, shipping tooling. Map them to the binaries and config keys your organization uses, and the workflow will translate directly.*

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.