Security Incident Response

Production engineering guide for security incident response covering patterns, implementation strategies, and operational best practices.

Security incident response is the disciplined sequence of steps taken from the moment a breach is suspected or confirmed through to final remediation and lessons learned. It activates when alerts cross a severity threshold, when indicators of compromise are validated, or when a stakeholder reports unexpected system behavior. This reference maps the concrete commands, configuration checks, and data-preservation patterns you execute at each phase, organized by what you’re trying to achieve rather than by product taxonomy.

Detect & Triage

The first goal is to confirm whether an incident has occurred, assess its scope, and preserve the state needed for later analysis. Use the following commands to inspect running state, recent logs, and network posture without altering anything.


systemctl status <service-name>

# Show the last 50 log lines for a service
journalctl -u <service-name> -n 50 --no-pager
# List processes that match a suspicious pattern, excluding the grep command itself
ps aux | grep -i <pattern> | grep -v grep
# Verify endpoint responsiveness and capture HTTP status, timing, and exit code
curl -s -o /dev/null -w "HTTP %{http_code} in %{time_total}s\n" -m 10 https://<host>/health
# Inspect all Docker containers, including stopped ones, with a compact format
docker ps -a --format "{{.ID}} {{.Status}} {{.Names}}"
# List Kubernetes pods across all namespaces with full node and IP information
kubectl get pods -A -o wide

Typical error text you’ll encounter during triage:

When examining container images or pod images, docker images -a and kubectl get images (if available) reveal what’s running. For database-backed services, psql -U <user> -d <db> -c "SELECT state, query FROM pg_stat_activity WHERE state='active'" helps identify long-running or anomalous queries. Note that psql output varies by version; the state column and query column are standard across PostgreSQL 12+.

Contain

Containment stops further damage while you assess impact. The aim is to isolate the affected component without destroying evidence. Use network-level isolation, process termination, or service disabling depending on the threat model.

# Drop traffic from a compromised IP at the host firewall; appends rule to the end of the chain
iptables -A INPUT -s <bad-ip> -j DROP
# Block a CIDR range using ufw (simpler host-based firewall)
ufw deny from <bad-cidr> to any
# Terminate a rogue process by PID, sending SIGKILL
kill -9 <PID>
# Stop a Docker container without removing it; sends SIGTERM, waits 10s default, then SIGKILL
docker stop <container-id>
# Force-delete a Kubernetes pod that is stuck in Terminating, bypassing the default grace period
kubectl delete pod <pod-name> --grace-period=0 --force
# Disable a service so it does not auto-start on reboot; does not stop a running instance
systemctl disable <service-name>

Confusion points that silently fail: systemctl disable only masks the service from auto-start; it does not stop a running instance. Always pair it with systemctl stop <service-name> if the service is active. Likewise, iptables -A appends a rule at the end of the chain, so if an earlier ACCEPT rule matches the same traffic, the drop may never be reached. Use iptables -I INPUT 1 -s <bad-ip> -j DROP to insert at the head, or verify rule order with iptables -L -n --line-numbers.

For cloud-managed load balancers or ingress controllers, containment often means updating health‑check thresholds or pulling the endpoint’s DNS record, but the underlying pattern remains: isolate the network surface, then freeze the process state.

Eradicate

Eradication removes the root cause—malicious binaries, compromised credentials, misconfigured secrets, or vulnerable code—while ensuring you don’t leave a persistence mechanism behind.

# Remove a suspicious package installed via the system package manager (Debian/Ubuntu)
apt-get purge <malicious-package>
# Delete a package installed with YUM or DNF (RHEL/CentOS/Fedora)
yum remove <malicious-package>
# Delete a Docker image that was used as a launchpad
docker rmi <image-id>
# Remove a secret from a Kubernetes secret store
kubectl delete secret <secret-name>
# Delete a file and its history that may contain embedded payloads
rm -rf /tmp/suspicious-payload*
# Reset a compromised credential by generating a new value and rotating it in the store
psql -c "ALTER USER <username> WITH PASSWORD '<new-strong-password>'"

A quietly failing pattern: rm -rf deletes the file name from the directory entry but may not overwrite the data blocks. On solid-state drives, the underlying flash may retain the data for wear-leveling purposes. If the payload was a binary executed from /tmp, the process image may already be in memory; killing the process (as above) and clearing the directory is a best-effort step. For true sanitization, boot from a clean medium and zero the disk.

When removing Kubernetes secrets, remember that kubectl delete secret only removes the API object; the underlying data may persist in etcd until garbage collection runs. If the secret held TLS keys, rotate the certificates independently of the secret deletion.

Recover

Recovery restores normal operations with a clean, verified state. The focus is on confirming integrity, restarting services, and validating that no residual indicators remain.

# Restart a service that was stopped for containment
systemctl restart <service-name>
# Run a Docker container with a fresh image and an automatic restart policy
docker run -d --restart=unless-stopped <image-tag>
# Apply a corrected Kubernetes deployment manifest
kubectl apply -f deployment.yaml
# Restore a PostgreSQL database from a known-clean backup
pg_restore -d <db-name> <backup-file>.backup
# Verify that all pods are in a Ready state after rollout
kubectl get pods -A -o status
# Check that no unexpected processes are running after restart
ps aux | grep -i <service> | grep -v grep

Error text you may see during recovery:

If a backup is suspect, cross-check checksums: sha256sum <backup-file> against the value recorded in your backup manifest. A mismatch means the backup is corrupted and you must fall back to an earlier, verified copy.

Post-Incident Activity

Post-incident work closes the loop, ensures accountability, and reduces the likelihood of recurrence. Treat this phase as equally important as containment or eradication.

# Export the full journalctl stream for a service during the incident window
journalctl -u <service-name> --since "<incident-start>" --until "<incident-end>" > /tmp/incident-logs.json
# Extract and format key events with jq for readability
jq -c '{time, message}' /tmp/incident-logs.json > /tmp/incident-summary.json
# Record the git history of configuration changes that may have contributed to the incident
git log --all --oneline --since "<incident-start>" --until "<incident-end>" > /tmp/config-changes.txt
# Generate a timeline table from timestamped log entries
awk '/<timestamp-pattern>/ {print $1, $2, $3, $4, $5}' /var/log/<app>/structured.log > /tmp/incident-timeline.tsv
# Store the cleaned analysis in a version-controlled incident report directory
mkdir -p /repo/incidents/<id>
cp /tmp/incident-summary.json /tmp/config-changes.txt /tmp/incident-timeline.tsv /repo/incidents/<id>/

Sharp edges you only learn after breaking in production:

Key Takeaways

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.