Automated Remediation | The Garnet Wiki

Automated remediation is the process of detecting system anomalies and applying corrective actions—without human intervention—through pre-defined, executable workflows. It matters most when systems grow too complex to monitor manually, and when downtime, configuration drift, or performance degradation must be resolved faster than human teams can react. The goal is to detect a failure, diagnose its root cause, and execute a fix before users notice, using tools that run continuously and respond to events.

Detecting Anomalies

Anomalies trigger remediation workflows. They originate from metrics, logs, traces, or direct health checks. The detection mechanism must be reliable, low-latency, and capable of distinguishing noise from real issues.

A Prometheus alert rule detects CPU load above 80% for 5 minutes across all web servers:

  groups:
    - name: web-servers
      rules:
        - alert: HighCPU
          expr: avg by(instance) (rate(node_cpu_seconds_total{mode="user"}[5m])) > 0.8
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "High CPU on {{ $labels.instance }}"
            description: "CPU usage has been above 80% for the last 5 minutes on {{ $labels.instance }}"

A Grafana dashboard triggers an alert when a service's error rate exceeds 2% over 10 minutes, with a threshold that adapts to daily traffic patterns.

A systemd unit monitors a service and logs a failure when the process exits with non-zero code:

  [Unit]
  Description=Web API Service
  After=network.target

  [Service]
  ExecStart=/usr/bin/node /app/server.js
  Restart=always
  RestartSec=3
  StandardOutput=journal
  StandardError=journal
  TimeoutStopSec=30
  SuccessExitStatus=0 1 2 3 4 5

  [Install]
  WantedBy=multi-user.target

When the service fails, journalctl captures the error:

journalctl -u web-api.service --since "2024-05-01 14:00:00" --no-pager

Output:

May 01 14:03:22 app01 web-api.service[1234]: Error: Cannot connect to PostgreSQL at db.internal:5432
May 01 14:03:22 app01 web-api.service[1234]:     at connect (/app/db.js:42:15)
May 01 14:03:22 app01 web-api.service[1234]:     at processTicksAndRejections (internal/process/task_queues.js:95:5)
May 01 14:03:22 app01 web-api.service[1234]: error: Could not connect to database

This error becomes the input to a remediation workflow.

Triggering Remediation Workflows

Once an anomaly is detected, the system must initiate a remediation workflow. The trigger must be consistent, idempotent, and include enough context to guide the fix.

A JSON event is published to a Kafka topic when a monitoring system detects a failure:

  {
    "event_id": "remediation-7a3b8c",
    "source": "prometheus",
    "alert_name": "HighCPU",
    "severity": "warning",
    "timestamp": "2024-05-01T14:03:22Z",
    "labels": {
      "instance": "web-01.prod",
      "job": "web-servers"
    },
    "annotations": {
      "summary": "High CPU on web-01.prod",
      "description": "CPU usage has been above 80% for the last 5 minutes on web-01.prod"
    },
    "metadata": {
      "duration": "5m",
      "value": 0.84
    }
  }

A curl command invokes a remediation API endpoint, passing the alert:

  curl -X POST https://remediation.api.example.com/v1/workflows \
       -H "Content-Type: application/json" \
       -d @remediation-event.json

The API returns a job ID and status:

  {
    "job_id": "job-8d4f2c",
    "status": "queued",
    "started_at": "2024-05-01T14:03:25Z"
  }

The workflow engine now owns the remediation process.

Executing Fix Actions

The core of automated remediation is the sequence of actions applied to resolve the anomaly. Each step must be idempotent, versioned, and retryable.

A remediation workflow is defined in YAML, using a standard orchestration format:

  workflow:
    name: "fix-high-cpu-on-web-server"
    description: "Reduce CPU load by restarting service and scaling out"
    steps:
      - name: "check-database-connectivity"
        action: "run-script"
        script: |
          #!/bin/bash
          set -e
          pg_isready -h db.internal -p 5432 -d appdb -U appuser --timeout=10
        retries: 3
        timeout: 30
        on_failure: "notify-incident"

      - name: "restart-web-service"
        action: "systemctl"
        command: "restart"
        service: "web-api.service"
        wait: true
        timeout: 60
        condition: "previous_step.success"

      - name: "scale-web-pods"
        action: "kubectl"
        command: "scale"
        resource: "deployment"
        name: "web-api"
        replicas: 4
        namespace: "production"
        condition: "check-database-connectivity.success"
        retryable: true

      - name: "verify-service-health"
        action: "http-request"
        method: "GET"
        url: "https://web-api.prod.example.com/health"
        expected_status: 200
        timeout: 20
        retries: 2
        on_success: "update-monitoring"

Each step is executed with a consistent interface:

systemctl restart web-api.service is always available.
kubectl scale deployment web-api --replicas=4 -n production is used across clusters.
curl -X GET https://web-api.prod.example.com/health returns a JSON response:

  {
    "status": "ok",
    "checks": {
      "database": "healthy",
      "cache": "healthy",
      "queue": "healthy"
    },
    "timestamp": "2024-05-01T14:05:45Z"
  }

Handling Failures in Remediation

Remediation workflows fail silently if not designed for resilience. Common failure modes include:

A step times out but is not retried.
A script exits with code 1, but the workflow treats it as success.
A kubectl command fails due to a missing --context, but no error is logged.

To mitigate this, workflows must:

Define explicit success and failure conditions.
Include retry logic with backoff.
Capture logs and output for each step.

For example:

- name: "restart-web-service"
  action: "systemctl"
  command: "restart"
  service: "web-api.service"
  wait: true
  timeout: 60
  condition: "previous_step.success"
  retries:
    count: 3
    delay: 10
    backoff: "exponential"
  on_failure:
    - "log-error"
    - "send-alert"
    - "rollback-deployment"

When a step fails, the system logs:

echo "Step 'restart-web-service' failed: systemctl returned exit code 1" >&2
journalctl -u web-api.service --since "2024-05-01 14:03:00" --no-pager

The failure is recorded in a central event store, such as a PostgreSQL table:

CREATE TABLE remediation_jobs (
  job_id UUID PRIMARY KEY,
  workflow_name TEXT NOT NULL,
  status TEXT NOT NULL,
  started_at TIMESTAMP WITH TIME ZONE,
  finished_at TIMESTAMP WITH TIME ZONE,
  error TEXT,
  step_history JSONB
);

INSERT INTO remediation_jobs (job_id, workflow_name, status, started_at)
VALUES ('job-8d4f2c', 'fix-high-cpu-on-web-server', 'running', '2024-05-01T14:03:25Z');

Propagating Results and Feedback

After execution, the workflow must report results and feed them back into the monitoring system.

The final step in a workflow updates Prometheus with a new metric:

  curl -X POST http://prometheus:9090/api/v1/write \
       -H "Content-Type: application/json" \
       -d '{
         "metric": {
           "__name__": "remediation_success_rate",
           "job": "web-api",
           "outcome": "fixed"
         },
         "values": [1.0],
         "timestamps": [1714585525000]
       }'

A notify-incident action sends a Slack message:

  {
    "text": "Remediation job <job-8d4f2c> succeeded after 2 minutes",
    "attachments": [
      {
        "title": "Details",
        "fields": [
          { "title": "Workflow", "value": "fix-high-cpu-on-web-server", "short": true },
          { "title": "Status", "value": "success", "short": true },
          { "title": "Duration", "value": "124s", "short": true },
          { "title": "Target", "value": "web-01.prod", "short": true }
        ],
        "color": "#2eb886"
      }
    ]
  }

A rollback-deployment step runs:

  kubectl scale deployment web-api --replicas=2 -n production --context=staging
  kubectl rollout undo deployment web-api -n production

The entire cycle is now closed: detection → trigger → execution → feedback.

Common Pitfalls and Sharp Edges

Confusing timeout and retries: A step may timeout after 30 seconds but be marked successful even if it completes in 15, leading to false confidence. The retries field must be tied to a retryable flag, and each retry must be logged.

Missing wait: true in systemctl steps: Without it, the service restarts in the background, but the next step runs before the service is ready. This causes the scale-web-pods step to fail because the web API is still booting.

Assuming kubectl is in the path: A workflow may fail silently if the kubectl binary is missing or not configured with a ~/.kube/config file. The --context and --namespace flags must be explicit.

Overlooking idempotency: A script that restarts a service but does not check if it’s already running may double the load. A simple systemctl start is not idempotent if the service is already active.

Ignoring step outputs: A run-script step returns a stdout and stderr, but the workflow engine uses only stdout. When a script fails with a detailed error in stderr, but stdout is empty, the error is lost.

Using on_failure but not on_success: A step may succeed but not update the monitoring system. on_success is not a default; it must be explicitly declared.

Failure to capture and correlate logs: A remediation job fails, but the logs are scattered across multiple systems. A job_id must be passed through every command:

  export REMEDIATION_JOB_ID=job-8d4f2c
  echo "Starting remediation job $REMEDIATION_JOB_ID"
  kubectl scale deployment web-api --replicas=4 -n production --context=prod \
    --dry-run=client -o yaml | tee /var/log/remediation/$REMEDIATION_JOB_ID.yaml

Monitoring and Maintenance

Remediation workflows are not set-and-forget. They require monitoring and periodic review.

A remediation-dashboard is built using Grafana, showing:
Number of remediation jobs per hour.
Success and failure rates.
Average duration per step.
Top failure reasons.

A check-remediation-status script runs daily:

  #!/bin/bash
  # Check for remediation jobs that failed but were not retried
  psql -c "
    SELECT job_id, workflow_name, error, started_at
    FROM remediation_jobs
    WHERE status = 'failed' AND retries_completed < max_retries
    ORDER BY started_at DESC
    LIMIT 10;
  " | mail -s "Remediation Failures Detected" [email protected]

A reconcile-remediation-state job compares expected state (e.g., 4 replicas for web-api) with actual state from Kubernetes and updates the remediation_jobs table.

Automated remediation is not just about fixing problems—it is about building a self-healing infrastructure where failure is invisible to users and detectable to engineers only when they look.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.