Automation Documentation

Production engineering guide for automation documentation covering patterns, implementation strategies, and operational best practices.

Automation Documentation is the deliberate, structured practice of recording *why* and *how* automation runs, not just *what* it does. It matters when a script, workflow, or pipeline fails, and someone—new or experienced—needs to understand, debug, or extend it without guessing. It is essential when automations evolve across teams, systems, and time, and when failures must be diagnosed across multiple layers of tooling.

Defining Automation Documentation

Automation documentation is not a single file. It is a collection of interlinked, living records—code comments, runbooks, configuration guides, error logs, and decision logs—that describe every aspect of an automation system. It answers: What does this automation do? Why was it built this way? How does it behave under stress? What does it mean when it fails? How can it be extended, monitored, and maintained?

It is most needed when:

The core of automation documentation is a decision log, a simple but powerful artifact that records design choices as they happen. It is written in a standard format: a list of changes, each with a timestamp, author, and a concise explanation of the choice. Each entry links to the code, configuration, or runbook that implements it.

2024-03-15 — johndoe
  • Added automated dependency update for Python services using pip-tools
  • Why: Reduces merge conflicts in requirements.txt and ensures consistent environments
  • Link: https://github.com/acme/infra-config/blob/main/ansible/roles/python-dependencies/tasks/main.yml
  • Trade-off: Increased CI time by 3 minutes; chosen over manual updates due to 10+ services using Python

This decision log is not a supplement—it is the spine of the documentation. It is written in real time, as changes are made.

Writing and Maintaining Automation Documentation

1. Documenting the Automation's Purpose and Scope

Every automation begins with a README.md file at the root of its repository. This file is not a placeholder. It contains:



Automates the validation of cluster configurations against security and operational standards.

## Overview

[Cluster] → [Kube-Config] → [Compliance Runner] → [Report] → [Slack, Jira]


### Components

| Component           | Responsibility                                  |
|---------------------|--------------------------------------------------|
| `check-ns.yaml`     | Validates namespace labels and annotations       |
| `check-policies.yaml` | Runs OPA policies against cluster state         |
| `reporter.py`       | Generates HTML report and uploads to S3          |

### Inputs

- `cluster-name`: string, e.g., `prod-us-east-1`
- `check-period`: string, e.g., `daily`, `weekly`

### Outputs

- `report-2024-03-15.html`
- `report-2024-03-15.json`
- Slack notification with summary and links

This README.md is the first point of contact. It is updated with every significant change.

2. Embedding Documentation in Code and Configuration

Automation code is documentation. But only when it is *written for humans*, not just machines.

In Ansible, every role includes a README.md that explains:

# roles/compliance-checks/defaults/main.yml
---
# Default values for compliance checks
compliance_checks_enabled: true
compliance_checks_interval: "1d"
compliance_checks_policies_dir: "/etc/compliance/policies"
compliance_checks_output_format: "json"
compliance_checks_notify_slack: true
compliance_checks_slack_webhook_url: "{{ lookup('env', 'SLACK_WEBHOOK_URL') }}"

The main.yml file includes inline comments that explain *why* a task is structured as it is.

- name: Run OPA policy evaluation
  opa:
    policy: "{{ item }}"
    input: "{{ cluster_state }}"
    output: "{{ compliance_results_dir }}/opa-{{ item }}.json"
  loop: "{{ compliance_checks_policies }}"
  # Why: OPA allows declarative policy logic and integrates well with Kubernetes
  # Why: Input is a full cluster state from kubectl get all -o json
  # Why: Output is versioned and used for audit trails

In Bash, every script includes a #-style header:

#!/bin/bash
# Automated Remediation for High CPU Usage
# Author: alice
# Last updated: 2024-03-10
# Runs every 15 minutes via systemd timer
#
# Inputs:
#   - $CLUSTER_NAME: name of Kubernetes cluster to monitor
#   - $THRESHOLD_CPU: CPU usage above which remediation triggers (default: 80)
#
# Outputs:
#   - Logs to /var/log/remediation.log
#   - Creates a Jira ticket on failure
#   - Updates Prometheus metric: remediation_last_run

This header is not just metadata—it is the first documentation a new engineer reads.

3. Managing Configuration and Environment

Configuration files are not just data. They are part of the documentation.

A config.yaml file for a CI pipeline is not just a list of values. It includes:

# config.yaml
# CI Pipeline: Automated Dependency Updates
# Hosted on GitHub Actions; runs on push to main and weekly

# How to update: 
# 1. Run `./scripts/update-dependencies.sh`
# 2. Commit changes and push to `dependencies` branch
# 3. PR opened automatically if changes detected

github:
  owner: "acme"
  repo: "infra-config"
  branch: "main"
  pr_template: "templates/pr-dependencies.md"

dependencies:
  python:
    tool: "pip-tools"
    requirements_in: "requirements.in"
    requirements_txt: "requirements.txt"
    update_interval: "weekly"
    max_changes: 5
    notify_on_failure: true
    slack_channel: "#infra-ci"
    # If not set, uses default from environment variable
    # If set to false, disables all notifications
    # If set to true, uses default Slack channel

  nodejs:
    tool: "npm"
    package_json: "package.json"
    update_interval: "daily"
    # If update_interval is missing, defaults to "daily"
    # If update_interval is "monthly", runs once per month
    # If update_interval is "on-push", runs on every push to main

The # comments are not optional—they are the primary documentation. They explain the intent behind the configuration.

4. Recording Automation Execution and Failure

Every automation produces logs. But logs are not documentation unless they are structured and annotated.

A standard format for logs is to use structured JSON, with a consistent set of keys across all automations.

{
  "timestamp": "2024-03-15T12:34:56Z",
  "automation": "compliance-checks",
  "run_id": "2024-03-15-01",
  "cluster": "prod-us-east-1",
  "status": "failed",
  "duration_ms": 4230,
  "inputs": {
    "check_period": "daily",
    "policies": [
      "security-baseline",
      "network-policy"
    ]
  },
  "outputs": {
    "report_url": "https://reports.acme.com/2024-03-15.html",
    "failed_checks": [
      "missing-namespace-label",
      "insecure-tls-config"
    ],
    "summary": "3 of 8 checks failed. 2 policies not applied."
  },
  "errors": [
    {
      "code": "OPA_EVAL_ERROR",
      "message": "OPA policy 'security-baseline' failed with errors",
      "details": [
        "Policy 'security-baseline' not found in OPA",
        "Input JSON missing 'metadata.annotations' field"
      ],
      "timestamp": "2024-03-15T12:34:56Z"
    },
    {
      "code": "SLACK_NOTIFY_FAILED",
      "message": "Failed to send Slack notification",
      "details": [
        "Slack webhook returned 400: Bad Request",
        "Payload: {\"text\": \"Compliance check failed\", \"channel\": \"#infra\"}"
      ],
      "timestamp": "2024-03-15T12:34:56Z"
    }
  ]
}

This structure allows tooling to extract and display key information. It also makes it easy to write custom dashboards and alerting rules.

5. Tracking Automation Evolution and Decisions

Every automation evolves. Each change should be recorded in a decision log, which is stored as a DECISIONS.md file in the project root.

Each decision is a new entry, with a unique ID, date, author, and a clear explanation.

# DECISIONS.md

## D-001: Choose OPA for compliance checks
2024-01-10 — johndoe
  • Problem: We need to validate cluster state against multiple policies, but current scripts are brittle and hard to extend.
  • Solution: Adopt Open Policy Agent (OPA) to define and evaluate policies declaratively.
  • Why OPA?
    - Policies are versioned and tested independently.
    - Integrates well with Kubernetes (via admission controller, gatekeeper, etc.).
    - Supports complex rule logic (e.g., nested conditions, data joins).
  • Trade-offs:
    - Increased complexity: new tooling, new data model.
    - Initial learning curve for team.
    - Decision: Accept complexity for long-term maintainability.
  • Links:
    - https://github.com/acme/infra-config/tree/main/opa
    - https://www.openpolicyagent.org/docs/latest/

This log is not a history—it is a *living record* of the automation’s evolution. It answers: Why did we build it this way?

Common Pitfalls and Silent Failures

1. The "Working" Automation That Is Actually Broken

An automation may run successfully—exit 0, status: passed—but still be wrong. The most common silent failure is misaligned inputs.

For example, a script expects cluster-name in config.yaml, but it is actually cluster_name. The automation runs, logs success, but applies checks to the wrong cluster.

# config.yaml
cluster-name: "staging-us-west-1"  # ← typo: should be 'cluster_name'

The script reads cluster_name but finds cluster-name. The key is not found. The default value is used, but the logs do not reflect this.

To catch such issues, every automation must include a validation step that checks input keys against expected values.

# Validate config before running
if ! grep -q "cluster_name" config.yaml; then
  echo "Warning: config.yaml missing key 'cluster_name'; using 'cluster-name' instead"
  # Use 'cluster-name' as fallback
  CLUSTER_NAME=$(yq '.cluster-name' config.yaml)
else
  CLUSTER_NAME=$(yq '.cluster_name' config.yaml)
fi

2. The Forgotten Dependency

An automation uses yq to parse YAML, but yq is not installed on the runner. The script runs, but fails silently because yq is missing.

The solution is to include a dependency check in every automation:

# Check for required tools
for tool in yq jq kubectl; do
  if ! command -v "$tool" &> /dev/null; then
    echo "Error: $tool is not installed. Please install it."
    exit 1
  fi
done

This check should be the first step in every automation.

3. The Misunderstood Configuration Key

A configuration key update_interval is used in many automations. But in one, update_interval is a string like "weekly", while in another, it is a number like 7. The automation expects a number but gets a string.

The key is not documented well enough.

The solution: define a schema for configuration keys.

# Configuration Schema: update_interval
# Type: string or integer
# Valid values:
#   - "daily", "weekly", "monthly", "on-push"
#   - integer: days between updates (e.g., 1 for daily, 7 for weekly)
# If string, maps:
#   - "daily" → 1
#   - "weekly" → 7
#   - "monthly" → 30
#   - "on-push" → null
# If integer, uses value as days.

This schema is linked from every configuration file and used in documentation.

Summary

Automation documentation is not a side activity. It is the core of automation. It is the bridge between code and understanding. It answers: *Why does this automation do what it does?* It is written in real time, embedded in code, and structured for discovery. It includes decision logs, configuration schemas, standardized logs, and clear READMEs. It handles the silent failures—misaligned inputs, missing tools, misinterpreted keys—and makes automation not just runnable, but understandable, maintainable, and resilient.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.