Toil Reduction Automation

Production-grade guide to toil reduction automation covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.

This reference describes how to automate repetitive engineering tasks that consume time and attention without delivering direct value. When teams spend more than 20% of their time on manual operations, process steps, or recovery actions, automation becomes a priority.

Automate Recovery Procedures

Create a script to restart failed services

When a service fails, manual intervention often takes longer than the actual fix. Use systemctl to create a restart script that also sends notifications:

#!/bin/bash
SERVICE_NAME="myapp"
systemctl restart "$SERVICE_NAME"
if [ $? -ne 0 ]; then
    echo "Failed to restart $SERVICE_NAME" >&2
    exit 1
fi

# Send notification via webhook
curl -X POST \
     -H 'Content-Type: application/json' \
     -d '{"service": "'"$SERVICE_NAME"'", "status": "restarted"}' \
     https://internal-notifications.example.com/webhook

When systemctl fails to restart a service, it returns exit code 1. If the service does not exist, systemctl exits with code 5. Automation must check these specific error codes.

Monitor and auto-restart containers

Use docker-compose with restart policies instead of manual restarts:

version: '3.8'
services:
  web:
    image: nginx:latest
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost"]
      interval: 30s
      timeout: 10s
      retries: 3

When a container fails its health check, docker-compose will restart it automatically. If the restart policy is "no", the container will not restart after failure.

Automate Configuration Changes

Use git hooks to validate configuration changes

Set up pre-commit hooks to validate that changes to configuration files are valid before they're committed:

#!/bin/bash
# .git/hooks/pre-commit
if git diff --cached --name-only | grep -E '\.(yaml|yml|json)$'; then
    # Validate config files
    for file in $(git diff --cached --name-only | grep -E '\.(yaml|yml|json)$'); do
        if ! python -m jsonschema -i "$file" schema.json 2>/dev/null; then
            echo "Invalid configuration in $file"
            exit 1
        fi
    done
fi

When a configuration change fails validation, the commit will be rejected. The error message is generated by the JSON schema validator and shows which field failed.

Deploy configuration changes with versioned manifests

Store configuration in git repositories and deploy with versioned Kubernetes manifests:

kubectl apply -f configmap.yaml --record
kubectl rollout status deployment/myapp

If --record is omitted, you lose visibility into what changed in the deployment history. The rollout status command returns exit code 1 if it times out or fails to complete.

Automate Incident Response

Create a runbook automation script

Use curl and jq to create incident response workflows that query internal systems:

#!/bin/bash
INCIDENT_ID="$1"
# Get incident details from internal API
INCIDENT=$(curl -s "https://internal-incident-system.example.com/incidents/$INCIDENT_ID")
# Extract service name
SERVICE=$(echo "$INCIDENT" | jq -r '.service')
# Check if service is healthy
HEALTH=$(curl -s "https://internal-monitoring.example.com/health/$SERVICE")
if [ "$(echo "$HEALTH" | jq -r '.status')" = "unhealthy" ]; then
    echo "Service $SERVICE is unhealthy, triggering restart"
    systemctl restart "$SERVICE"
fi

When the incident system returns a 404 for an unknown incident ID, curl exits with code 22. The JSON parsing may fail silently if the response format changes.

Automate data backup and restore operations

Use rsync with logging to automate backups:

#!/bin/bash
BACKUP_DIR="/backups"
SOURCE_DIR="/data"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
BACKUP_NAME="backup-${TIMESTAMP}.tar.gz"

tar -czf "$BACKUP_DIR/$BACKUP_NAME" "$SOURCE_DIR"
if [ $? -ne 0 ]; then
    echo "Backup failed at $(date)" >&2
    exit 1
fi

# Verify backup integrity
gzip -t "$BACKUP_DIR/$BACKUP_NAME"
if [ $? -ne 0 ]; then
    echo "Backup verification failed" >&2
    exit 1
fi

When tar fails to create the archive, it returns exit code 1. If gzip cannot verify the file, it returns exit code 1. The backup script must check both operations.

Automate Deployment Processes

Create automated deployment pipelines

Use git hooks or CI/CD tools to automate deployments:

#!/bin/bash
# Pre-deployment checks
if ! git diff --cached --name-only | grep -E '\.(js|ts|css)$'; then
    echo "No frontend changes, skipping build"
    exit 0
fi

# Build and deploy
npm run build
if [ $? -ne 0 ]; then
    echo "Build failed" >&2
    exit 1
fi

# Deploy to staging
kubectl set image deployment/myapp web=nginx:latest --namespace staging

If the npm build command fails, it returns exit code 1. The kubectl command will return 1 if it cannot find the deployment or namespace. When a deployment fails, kubectl does not automatically rollback.

Automate rollbacks

Create rollback scripts that can revert deployments:

#!/bin/bash
# Rollback to previous version
kubectl rollout undo deployment/myapp --to-revision=1
if [ $? -ne 0 ]; then
    echo "Rollback failed" >&2
    exit 1
fi

# Wait for deployment to complete
kubectl rollout status deployment/myapp

The rollback command fails if the revision does not exist. If the rollout status times out, it returns exit code 1. Rollbacks do not automatically notify stakeholders.

Automate Monitoring and Alerting

Create automated alert resolution scripts

Use a combination of curl and shell scripting to handle alerts:

#!/bin/bash
ALERT_ID="$1"
# Acknowledge alert
curl -X POST \
     -H 'Content-Type: application/json' \
     -d '{"status": "acknowledged"}' \
     "https://internal-alerting.example.com/alerts/$ALERT_ID"

# Run diagnostics
if ! systemctl is-active --quiet myapp; then
    systemctl restart myapp
fi

# Send notification
curl -X POST \
     -H 'Content-Type: application/json' \
     -d '{"alert": "'"$ALERT_ID"'", "action": "restarted"}' \
     https://internal-notifications.example.com/webhook

If the alerting system returns HTTP 404 for an unknown alert ID, the script fails. If systemctl cannot restart a service, it exits with code 1. The script does not check if the service actually starts after restart.

Automate log analysis and alert generation

Use grep and awk to analyze logs automatically:

#!/bin/bash
LOG_FILE="/var/log/myapp.log"
ERRORS=$(grep -c "ERROR\|FATAL" "$LOG_FILE")
if [ "$ERRORS" -gt 10 ]; then
    curl -X POST \
         -H 'Content-Type: application/json' \
         -d '{"message": "High error rate detected", "count": '"$ERRORS"'"}' \
         https://internal-alerting.example.com/webhook
fi

When grep cannot read the log file, it returns exit code 2. If the alerting webhook is unreachable, curl exits with code 7. The script does not handle log rotation or log file permissions.

Common Automation Patterns and Failures

Handle timeouts in automation

Always set explicit timeouts for long-running operations:

timeout 60s kubectl rollout status deployment/myapp
if [ $? -eq 124 ]; then
    echo "Rollout timed out after 60 seconds"
    exit 1
fi

When a command times out, timeout returns exit code 124. If the timeout is not explicitly set, automation can hang indefinitely.

Validate automation outputs

Always validate that automation has succeeded:

#!/bin/bash
kubectl apply -f deployment.yaml
if [ $? -ne 0 ]; then
    echo "Deployment failed" >&2
    exit 1
fi

# Wait for deployment to be ready
kubectl wait --for=condition=available deployment/myapp --timeout=60s
if [ $? -ne 0 ]; then
    echo "Deployment not ready within timeout" >&2
    exit 1
fi

The kubectl wait command returns exit code 1 if the condition is not met. If the deployment YAML has syntax errors, apply will fail but not provide detailed information about which field failed.

Test automation scripts in isolation

Create a test script to validate that your automation works:

#!/bin/bash
# Test backup script
echo "Testing backup script..."
./backup.sh
if [ $? -ne 0 ]; then
    echo "Backup test failed"
    exit 1
fi

# Verify backup exists
if [ ! -f "/backups/backup-$(date +%Y%m%d-%H%M%S).tar.gz" ]; then
    echo "Backup file not found"
    exit 1
fi

When the backup script fails, it must return appropriate exit codes. If a test fails to validate that a backup was created, the automation pipeline does not catch this failure.

Automation Failure Modes

Silent failures in service restarts

When systemctl cannot restart a service, it may silently fail and return exit code 0:

# This may succeed even though restart failed
systemctl restart non-existent-service
echo $? # Will show 0 even though service does not exist

Always check systemctl's output for specific error messages rather than relying solely on exit codes.

Configuration validation failures

When configuration validation fails, the script should exit with appropriate error codes:

# This will silently continue if validation fails
python -m jsonschema -i config.json schema.json 2>/dev/null

# Better approach:
if ! python -m jsonschema -i config.json schema.json; then
    echo "Configuration validation failed"
    exit 1
fi

Resource cleanup failures

When automation needs to clean up resources, it should handle cleanup failures:

# Clean up temporary files
trap 'rm -f /tmp/tempfile.XXXXXX' EXIT
# If cleanup fails, the script should still report it
if ! rm -f /tmp/tempfile.XXXXXX; then
    echo "Cleanup failed" >&2
fi

Notification failures in automation

When notifications fail, automation must not assume success:

# Notification may fail without failing the entire script
curl -X POST -d '{"status": "deployed"}' https://notifications.example.com/endpoint
if [ $? -ne 0 ]; then
    echo "Notification failed but deployment continues"
fi

Automation scripts must be designed to handle partial failures gracefully.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.