Automation Error Handling

Production engineering guide for automation error handling covering patterns, implementation strategies, and operational best practices.

Automation Error Handling is the practice of detecting, diagnosing, and recovering from failures in automated systems—whether those are deployment pipelines, configuration runs, or recurring maintenance tasks. It matters when a single automation step fails, but the entire workflow collapses silently, leaving teams unaware until metrics degrade or users report issues. This page details how to build robust error handling into automation, from the first command to the final alert, using universally applicable patterns across real, existing tools.

Detecting Errors in Automation

Automation must know when something has gone wrong. A task that completes without error is not necessarily correct. The first step is to define what constitutes a failure.

Most tools report exit codes: 0 means success, any non-zero value indicates failure. The common convention is that 1 is a general error, 2 a misused command, 3 a configuration issue, and so on. But these are not universally observed.

In a shell script or a CI job, a command like kubectl apply -f manifests/ may succeed but fail to deploy a service due to a timeout or a missing image. The script continues, but the error goes unnoticed.

To detect such failures, use set -e in Bash scripts to halt execution on any non-zero exit code:

#!/bin/bash
set -e

kubectl apply -f manifests/
helm upgrade --install myapp ./charts/myapp
./scripts/validate-deployment.sh

The set -e flag ensures that if helm upgrade returns 1, the script stops immediately, preventing cascading failures.

For more granular control, combine set -euo pipefail:

#!/bin/bash
set -euo pipefail

export KUBECONFIG=~/.kube/config
export ENVIRONMENT="${ENVIRONMENT:-production}"

kubectl get pods -n "$ENVIRONMENT" | grep Running

If kubectl get pods fails, or grep Running returns no output, the pipeline fails and the script exits with the correct error code.

When orchestrating workflows across multiple services, use jq to parse JSON output and check for specific failure conditions:

#!/bin/bash
set -euo pipefail

STATUS_JSON=$(curl -s http://api.example.com/status)

if ! echo "$STATUS_JSON" | jq -e '.status == "error" and .error.code == 404'; then
  echo "API is down or returned 404"
  exit 1
fi

The jq -e flag causes jq to return 1 if the expression fails, enabling tight integration into the exit code pipeline.

Reporting and Logging Errors

Once an error is detected, it must be reported. Logging is not just for debugging; it is the foundation for diagnosing automation failures.

Use structured logging with consistent formats. JSON logs are preferred for their machine-readability and ease of parsing.

In a Python automation script, emit logs using logging with json format:

import logging
import json

logging.basicConfig(
    level=logging.INFO,
    format='{"timestamp": "%(asctime)s", "level": "%(levelname)s", "message": "%(message)s", "task": "%(task_id)s"}',
    handlers=[
        logging.FileHandler("/var/log/automation/run.log"),
        logging.StreamHandler(),
    ],
)

logger = logging.getLogger("automation")

def deploy_service(service_name, version):
    logger.info(f"Starting deployment of {service_name} v{version}", extra={"task_id": "deploy-123"})
    try:
        # Simulate deployment
        result = subprocess.run(
            ["kubectl", "apply", "-f", f"manifests/{service_name}.yaml"],
            check=True,
            capture_output=True,
            text=True,
        )
        logger.info("Deployment succeeded", extra={"output": result.stdout})
    except subprocess.CalledProcessError as e:
        logger.error(
            "Deployment failed",
            extra={
                "error_code": e.returncode,
                "stderr": e.stderr,
                "stdout": e.stdout,
                "task_id": "deploy-123",
            },
        )
        raise

This produces logs like:

{"timestamp": "2024-03-15T12:34:56.789Z", "level": "INFO", "message": "Starting deployment of user-service v1.2.0", "task": "deploy-123"}
{"timestamp": "2024-03-15T12:35:10.123Z", "level": "INFO", "message": "Deployment succeeded", "output": "service/user-service unchanged", "task": "deploy-123"}
{"timestamp": "2024-03-15T12:35:15.456Z", "level": "ERROR", "message": "Deployment failed", "error_code": 1, "stderr": "error: unable to recognize \"manifests/user-service.yaml\": no matches for kind \"Service\" in version \"v1\"", "stdout": "", "task": "deploy-123"}

When an automation pipeline runs across multiple tools, collect logs from all steps into a central system. Use journalctl for systemd services, and docker logs for containerized tasks.

For example, after running docker-compose up -d, collect logs from all services:

#!/bin/bash
set -euo pipefail

docker-compose up -d


for service in $(docker-compose config --services); do
  docker logs --since "5m" "$service" | tee -a /var/log/automation/compose.log
done

This captures not only the output of the current run but also historical context, helping to diagnose intermittent failures.

Handling Errors: Retry, Skip, and Rollback

Automation is not just about detecting failure—it is about responding to it.

Retry Failed Steps

Automated tasks often fail transiently: a network timeout, a slow database, a temporary resource shortage. The solution is to retry.

Use while loops with exponential backoff:

#!/bin/bash
set -euo pipefail

RETRIES=5
BACKOFF=1
MAX_BACKOFF=60

for i in $(seq 1 $RETRIES); do
  if kubectl rollout status deployment/frontend --timeout=30s; then
    echo "Deployment succeeded"
    exit 0
  fi

  echo "Attempt $i failed. Retrying in $BACKOFF seconds..."
  sleep $BACKOFF

  BACKOFF=$((BACKOFF * 2))
  if [ $BACKOFF -gt $MAX_BACKOFF ]; then
    BACKOFF=$MAX_BACKOFF
  fi
done

echo "All retries exhausted. Deployment failed."
exit 1

This handles common transient errors, such as a pod taking longer than expected to start.

For more complex scenarios, use curl to retry a failed API call with a backoff strategy:

#!/bin/bash
set -euo pipefail

API_URL="https://api.example.com/v1/deployments"
DEPLOYMENT_ID="d-12345"

for attempt in $(seq 1 10); do
  response=$(curl -s -w "%{http_code}" -o /tmp/response.json "$API_URL/$DEPLOYMENT_ID")

  http_code=$(tail -n1 /tmp/response.json)
  body=$(head -n -1 /tmp/response.json)

  if [ "$http_code" -eq 200 ]; then
    echo "Success: $body"
    exit 0
  fi

  if [ "$http_code" -ge 500 ] || [ "$http_code" -eq 429 ]; then
    echo "Server error or rate limited. Retrying in $((attempt * 10)) seconds..."
    sleep $((attempt * 10))
    continue
  fi

  echo "Client error: $body"
  exit 1
done

echo "All retries failed."
exit 1

The script adapts the retry delay based on the number of attempts and distinguishes between server and client errors.

Skip or Continue on Failure

Sometimes, a failure is non-critical. The automation can proceed despite a problem.

Use || to continue after a command, and && to require success.

#!/bin/bash
set -euo pipefail

# Deploy the main application
kubectl apply -f manifests/app.yaml || echo "App deployment failed but continuing..."

# Run migration, but don’t stop if it fails
./scripts/migrate-db.sh && echo "Database migration completed" || echo "Migration failed but continuing..."

# Deploy sidecar services only if app deployment succeeded
if kubectl get deployment app; then
  kubectl apply -f manifests/sidecar.yaml
else
  echo "App deployment failed. Skipping sidecar deployment."
fi

This pattern allows for graceful degradation: critical steps block the pipeline, while non-critical ones continue in the background.

Rollback on Failure

When a deployment fails, roll back to the previous known good state.

Use kubectl rollout undo to revert a deployment:

#!/bin/bash
set -euo pipefail

DEPLOYMENT_NAME="frontend"
PREVIOUS_REVISION=$(kubectl rollout history deployment/$DEPLOYMENT_NAME --revision=1 --output=json | jq -r '.[0].revision')

if ! kubectl rollout status deployment/$DEPLOYMENT_NAME --timeout=60s; then
  echo "Rolling back to revision $PREVIOUS_REVISION"
  kubectl rollout undo deployment/$DEPLOYMENT_NAME --to-revision=$PREVIOUS_REVISION
  kubectl rollout status deployment/$DEPLOYMENT_NAME --timeout=60s
fi

This ensures that if a new version fails, the system reverts to the last stable version, minimizing downtime.

For database migrations, use a two-phase rollback:

#!/bin/bash
set -euo pipefail

# Apply migration
./scripts/migrate-db.sh

if ! psql -c "SELECT 1 FROM schema_migrations WHERE version = '202403151200';" -d myapp; then
  echo "Rollback: failed to apply migration. Rolling back..."
  ./scripts/rollback-db.sh
  # Check rollback success
  if ! psql -c "SELECT 1 FROM schema_migrations WHERE version = '202403101200';" -d myapp; then
    echo "Rollback failed. Sending alert."
    curl -X POST -H "Content-Type: application/json" \
      -d '{"title": "Database Rollback Failed", "message": "Failed to revert to version 202403101200"}' \
      http://alerting.example.com/api/v1/alerts
    exit 1
  fi
fi

The script checks that the rollback succeeded by verifying the presence of the expected schema version.

Alerting on Automation Failures

When an automation step fails, the system must notify the right people.

Use curl to send alert notifications to an HTTP endpoint:

#!/bin/bash
set -euo pipefail

ALERT_URL="http://alerting.example.com/api/v1/alerts"

send_alert() {
  local title="$1"
  local message="$2"
  local severity="${3:-warning}"

  curl -X POST -H "Content-Type: application/json" \
    -d "{
      \"title\": \"$title\",
      \"message\": \"$message\",
      \"severity\": \"$severity\",
      \"source\": \"automation\",
      \"timestamp\": \"$(date -Iseconds)\"
    }" \
    "$ALERT_URL"
}

# In a workflow
if ! kubectl apply -f manifests/app.yaml; then
  send_alert "Deployment Failed", "Failed to apply app manifest", "critical"
  exit 1
fi

This produces structured alerts that can be consumed by monitoring tools like Prometheus, Grafana, or Slack.

For real-time alerts, use slack as a target:

send_slack_alert() {
  local title="$1"
  local message="$2"
  local webhook_url="$3"
  local color="${4:-#ff6600}"

  curl -X POST -H "Content-Type: application/json" \
    -d "{
      \"text\": \"*${title}*\",
      \"attachments\": [{
        \"color\": \"$color\",
        \"text\": \"$message\",
        \"ts\": $(date +%s)
      }]
    }" \
    "$webhook_url"
}

This format integrates seamlessly with existing alerting infrastructure.

Failure Modes and Silent Pitfalls

Not all errors are visible. Some silently fail, and only years of production use expose them.

Common mistakes:

Another silent pitfall: timestamp drift. Logs from different services use different time zones or system clocks. Use iso8601 timestamps in logs and alerting systems to ensure consistency.

Finally, error propagation across services often fails. A failure in one automation step is logged but not surfaced to the next. Use a central error registry, such as a failed_jobs table in PostgreSQL:

CREATE TABLE failed_jobs (
  id SERIAL PRIMARY KEY,
  job_name TEXT NOT NULL,
  error_message TEXT NOT NULL,
  metadata JSONB,
  created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
  retried_at TIMESTAMP WITH TIME ZONE
);

Each automation step inserts a record on failure:

#!/bin/bash
set -euo pipefail

insert_error() {
  local job_name="$1"
  local error_message="$2"
  local metadata_json="$3"

  psql -c "
    INSERT INTO failed_jobs (job_name, error_message, metadata)
    VALUES ('$job_name', '$error_message', '$metadata_json')
  "
}

# In a workflow
if ! ./scripts/backup-database.sh; then
  insert_error "backup-database", "Backup failed", '{"duration": "120s", "size": "2.1GB"}'
  exit 1
fi

This creates a persistent audit trail of automation failures, enabling deeper analysis and proactive monitoring.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.