Security Incident Response

Plan, execute, and improve security incident response. Covers incident response frameworks, detection engineering, containment strategies, forensics, communication plans, and the after-action review process that turns incidents into organizational learning.

Security Incident Response is the process of detecting, analysing, containing, eradicating, and recovering from a security breach or cyberattack. It matters when a system shows signs of compromise—unexpected network traffic, failed logins, file modifications, or alerts from monitoring tools—and the engineering team must act rapidly, systematically, and predictably to limit damage and restore trust.

Detection and Triage

An incident begins when a signal triggers a response. The earliest signal is often an alert from a SIEM (Security Information and Event Management) system, but it can be a log entry, a health check failure, or a custom metric crossing a threshold. The first action is to collect and verify the signal.

For example, a kubernetes cluster generates a pod-creation event with an unusual container-image field:

{
  "timestamp": "2023-10-05T12:34:56Z",
  "event": "pod-creation",
  "namespace": "prod",
  "pod": "web-7c8d9f1234",
  "image": "registry.example.com/attacker-backdoor:latest",
  "host": "worker-3.prod.example.com"
}

This event is picked up by a monitoring system via a curl-based alerting rule:

curl -X POST https://alerting.example.com/api/v1/alert \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Unexpected Pod Image",
    "description": "Pod created with image not in known baseline",
    "severity": "high",
    "timestamp": "2023-10-05T12:34:56Z",
    "data": {
      "pod": "web-7c8d9f1234",
      "image": "registry.example.com/attacker-backdoor:latest"
    }
  }'

The alert is ingested into a central incident tracker. The first triage task is to validate the signal by querying related systems.

Validate the Signal

To confirm the alert is not a false positive, the responder queries the container-image registry:

curl -H "Authorization: Bearer $REGISTRY_TOKEN" \
  https://registry.example.com/v2/attacker-backdoor/manifests/latest \
  | jq '.layers[].digest'

This returns a list of layer digests, which can be used to inspect the image contents:

docker pull registry.example.com/attacker-backdoor:latest
docker inspect registry.example.com/attacker-backdoor:latest

The output reveals a Dockerfile with a RUN command that installs a custom backdoor.py script and sets up a listener on port 8080. The image is not part of the known CI/CD pipeline, and the backdoor.py script is not signed or versioned.

The responder logs into the worker-3 host and checks the pod process:

ps aux | grep -E 'python.*backdoor.py'

This shows a Python process running with --listen 8080 --keyfile /etc/secrets/backdoor.key, indicating the backdoor is active.

Analysis and Investigation

With the signal validated, the next phase is to determine the scope and root cause of the incident.

Map the Attack Surface

The responder queries the kubernetes API to find all pods using the suspicious image:

kubectl get pods -n prod -o wide \
  --field-selector=status.phase=Running \
  --show-labels \
  | grep "attacker-backdoor"

This returns:

NAME           READY   STATUS    RESTARTS   AGE   NODE
web-7c8d9f1234   1/1     Running   0          2h    worker-3.prod.example.com
api-1a2b3c4d5e   1/1     Running   1          4h    worker-1.prod.example.com

The backdoor is not isolated to one pod. The api-1a2b3c4d5e pod has been running for 4 hours and has a ConfigMap entry with a secret-key field.

The responder downloads the ConfigMap:

kubectl get configmap -n prod api-config -o yaml

The ConfigMap contains:

apiVersion: v1
kind: ConfigMap
metadata:
  name: api-config
  namespace: prod
data:
  secret-key: "a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6"
  log-level: "debug"
  backdoor: "enabled"

The secret-key is not in the vault system, and the log-level is set to debug, suggesting the team has been logging extensively.

Correlate Logs and Events

The responder collects logs from all affected pods and correlates them with system-level events.

They fetch logs from worker-3:

journalctl -u kubelet -u containerd --since "2023-10-05T12:00:00Z" \
  --output=short-precise \
  --no-pager

This shows:

Oct 05 12:34:56 worker-3 kubelet[1234]: Pod "web-7c8d9f1234" created
Oct 05 12:35:01 worker-3 containerd[5678]: Pulling image registry.example.com/attacker-backdoor:latest
Oct 05 12:35:10 worker-3 python[9101]: Starting backdoor server on 0.0.0.0:8080
Oct 05 12:35:12 worker-3 python[9101]: Listening for connections...

The logs confirm that the backdoor started shortly after the pod was created.

The responder queries the pg database for entries related to the secret-key:

SELECT id, user_id, created_at, metadata FROM access_logs
WHERE metadata->>'secret_key' = 'a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6'
  AND created_at > '2023-10-05T12:30:00Z';

The query returns 12 rows, all from 2023-10-05T12:35:00Z to 12:35:15Z, indicating a burst of activity coinciding with the backdoor startup.

Containment and Eradication

The incident is now understood: an attacker deployed a custom backdoor image, which is active on two pods, and the attacker has used the secret-key to authenticate and access the API.

Contain the Threat

The responder isolates the attack surface by:

  1. Adding a firewall rule to block incoming traffic to the backdoor on port 8080:
iptables -A INPUT -p tcp --dport 8080 -j DROP
  1. Labeling all pods using the suspect image:
apiVersion: v1
kind: Pod
metadata:
  name: web-7c8d9f1234
  namespace: prod
  labels:
    incident: "backdoor-20231005"
    severity: "high"
    owner: "security-team"
  1. Disabling the backdoor feature in the ConfigMap:
data:
  backdoor: "disabled"
  log-level: "info"
  1. Rotating the secret-key in the vault system:
vault write secret/prod/api/keys/current \
  value="new-secret-0x1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d"

Eradicate the Root Cause

The responder removes the attacker’s artifacts from the system.

  1. Deleting the backdoor image from the registry:
curl -X DELETE \
  -H "Authorization: Bearer $REGISTRY_TOKEN" \
  https://registry.example.com/v2/attacker-backdoor/manifests/latest
  1. Removing the backdoor.py script from all pods:
kubectl exec -n prod web-7c8d9f1234 -- rm /opt/backdoor.py
kubectl exec -n prod api-1a2b3c4d5e -- rm /opt/backdoor.py
  1. Patching the Dockerfile to include a healthcheck:
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD curl -f http://localhost:8080/health || exit 1
  1. Adding a cron job to scan for suspicious image values:

#!/bin/sh
for image in $(kubectl get pods -n prod -o jsonpath='{.items[*].spec.containers[*].image}'); do
  if ! grep -q "$image" /etc/known-images.txt; then
    echo "Suspicious image: $image" | mail -s "Incident Alert" [email protected]
  fi
done

Recovery and Post-Incident Review

The incident is declared resolved when all systems are stable, the threat is eradicated, and the team has documented the actions taken.

Restore Services

The responder updates the ConfigMap to re-enable the backdoor:

data:
  backdoor: "enabled"
  log-level: "info"
  secret-key: "new-secret-0x1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d"

They deploy the updated ConfigMap:

kubectl apply -f configmap/api-config.yaml

The api-1a2b3c4d5e pod restarts, and the backdoor service comes online with the new secret-key.

Document the Incident

The responder writes a detailed incident report in a standard format.

# Incident Report: Backdoor Deployment in Production

## Summary
An attacker deployed a custom backdoor image `registry.example.com/attacker-backdoor:latest` into the production `kubernetes` cluster, activating a listener on port 8080. The backdoor was used to authenticate and access the API service via a `secret-key`.

## Timeline
- 12:30:00: Pod `web-7c8d9f1234` created with new image.
- 12:35:00: Image pulled and backdoor started.
- 12:35:12: First incoming connection from attacker.
- 12:36:00: `secret-key` rotated and backdoor disabled.

## Actions Taken
- Blocked port 8080 via `iptables`.
- Updated `ConfigMap` with new `secret-key`.
- Removed backdoor script from all pods.
- Added `healthcheck` to `Dockerfile`.
- Created cron job for image monitoring.

## Lessons Learned
- The `ConfigMap` should be versioned and validated.
- `image` changes should trigger a `CI` pipeline to build and sign the image.
- `secret-key` rotation should be automated.

Automate the Response

The responder creates a playbook to automate future incidents of this type.

# incident-backdoor.yml
- name: Detect and Respond to Backdoor Deployment
  hosts: all
  tasks:
    - name: Find pods with suspicious image
      shell: |
        kubectl get pods -n prod -o wide \
          --field-selector=status.phase=Running \
          --show-labels \
          | grep "{{ image }}"
      register: pods
      vars:
        image: "registry.example.com/attacker-backdoor:latest"

    - name: Label affected pods
      shell: |
        kubectl label pods -n prod {{ item }} \
          incident=backdoor-{{ ansible_date_time.iso8601 }} \
          severity=high
      loop: "{{ pods.stdout_lines }}"

    - name: Disable backdoor
      shell: |
        kubectl patch configmap -n prod api-config \
          -p '{"data":{"backdoor":"disabled"}}'
      run_once: true

    - name: Rotate secret key
      shell: |
        vault write secret/prod/api/keys/current \
          value="{{ lookup('password', 'length=32') }}"
      run_once: true

    - name: Notify security team
      mail:
        to: [email protected]
        subject: "Backdoor Detected: {{ image }}"
        body: |
          A backdoor image has been detected in production.
          Affected pods: {{ pods.stdout_lines | join(', ') }}
          Backdoor disabled and secret key rotated.

This playbook runs on ansible and is triggered by git hooks or cron.

Common Pitfalls

Silent Failures

Confused Options

Sharp Edges

  curl -X POST https://alerting.example.com/api/v1/alert \
    -d '{ "name": "Backdoor Detected" }'

The alerting system expects application/json, but the Content-Type header is not set. The POST succeeds, but the alert is not processed. Adding -H "Content-Type: application/json" is critical.

  "Entrypoint": "/opt/backdoor.py"

The Entrypoint is a string, not a list, so the backdoor.py script is not executed as a command. The Dockerfile must use:

  ENTRYPOINT ["/opt/backdoor.py"]

instead of ENTRYPOINT /opt/backdoor.py.

The python backdoor.py --listen 8080 command is truncated in the ps output. Using ps auxww ensures full command lines are visible.

The process of Security Incident Response is not a checklist, but a continuous loop of detection, analysis, containment, eradication, and recovery—each step revealing deeper structure and more subtle failures.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.