Ansible for Infrastructure Automation: Playbooks That Do Not Break at 3 AM

Infrastructure automation is not just about deploying servers and services — it’s about ensuring that every change, from a new app release to a security patch, applies consistently, reliably, and without human error, even when the team is asleep. Ansible for Infrastructure Automation: Playbooks That Do Not Break at 3 AM is the practice of writing Ansible playbooks that, once deployed, survive the chaos of production: overnight rollouts, cascading failures, and the silent creep of misconfigured systems — all without requiring an engineer to wake up and fix things manually.

Designing Playbooks for Production Resilience

When a playbook runs at 3 AM and fails, the entire system is at risk. The ideal playbook doesn’t just run — it logs, monitors, and recovers itself. This section covers the core practices to build playbooks that are both correct and resilient.

Idempotency and State Management

Every Ansible task must be idempotent: running it multiple times produces the same outcome as running it once. This is not just a best practice — it’s a requirement.

The state parameter on copy, file, template, and package tasks is central to this. For example:

- name: Ensure nginx config is in place
  copy:
    src: /tmp/nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    owner: root
    group: root
    mode: '0644'
    backup: yes
    validate: 'nginx -t -c {{ item }}'
  loop:
    - /etc/nginx/nginx.conf
  register: nginx_config

The backup: yes option automatically creates a .backup file before updating, and validate runs a command after copy, failing the task if the configuration is invalid. The register step captures the result for later use.

Failures here are silent but catastrophic. A missing validate command will not fail the task if nginx -t returns success but the config is incomplete. The real failure mode is when validate runs but nginx -t exits with code 1 and no output. To catch this, the validate command must include --check and --error-log-path:

validate: |
  nginx -t -c {{ item }} --check --error-log-path=/var/log/nginx/validate.error.log

This ensures that even when nginx.conf is syntactically correct but missing a required server block, the failure is visible and actionable.

Variable Management and Configuration Hierarchy

Variables in Ansible are not just a list — they are a system. The order of resolution is critical:

1. host_vars/<hostname>.yml
2. group_vars/<groupname>.yml
3. group_vars/all.yml
4. vars/main.yml in roles
5. ansible-playbook command-line variables

A common mistake is assuming that group_vars/all.yml overrides host_vars/, but it does so only when the variable is not defined at a higher level. Use set_fact to surface configuration decisions:

- name: Determine environment-specific settings
  set_fact:
    app_version: "{{ lookup('file', '/etc/app/version') | default('latest') }}"
    log_level: "{{ 'debug' if env == 'development' else 'info' }}"
    backup_retention: "{{ 30 if env == 'production' else 7 }}"
  vars:
    env: "{{ lookup('env', 'ENVIRONMENT') | default('staging') }}"

The lookup plugin reads from the environment or a file, and set_fact creates new variables that can be used in later tasks. A failure here is a missing ENVIRONMENT variable — the playbook runs, but app_version is latest even when a version file exists. The fix is to add | default('latest') to the lookup call.

Error Handling and Failure Recovery

A playbook that does not break at 3 AM must handle failure gracefully. This is achieved through task-level and playbook-level error handling.

Task-Level Recovery

Use rescue blocks within tasks to define fallback logic for when a task fails.

- name: Deploy application
  block:
    - name: Install dependencies
      pip:
        requirements: /app/requirements.txt
        virtualenv: /app/venv
        virtualenv_command: /app/venv/bin/python -m venv
    - name: Run migrations
      command: /app/venv/bin/python manage.py migrate
      args:
        chdir: /app
  rescue:
    - name: Rollback to previous app version
      command: git checkout HEAD~1
      args:
        chdir: /app
    - name: Restart application
      systemd:
        name: app.service
        state: restarted
        enabled: yes
    - name: Notify on failure
      mail:
        to: "[email protected]"
        subject: "Deployment failed: {{ inventory_hostname }}"
        body: |
          Deployment failed on {{ inventory_hostname }}.

          Last known good commit: {{ git_head }}
          Error output:
          {{ ansible_playbook_results | to_nice_json }}

This structure ensures that if pip install fails, the system rolls back to the previous commit and restarts the service. The mail task uses ansible_playbook_results — the full output of the playbook execution — to provide context.

Playbook-Level Error Handling

Use handlers and meta: flush to coordinate recovery across multiple hosts.

- name: Deploy and configure web servers
  hosts: webservers
  become: true
  vars:
    nginx_config: "{{ lookup('file', '/tmp/nginx.conf') }}"
  roles:
    - nginx
    - app
  handlers:
    - name: Reload nginx
      systemd:
        name: nginx
        state: reloaded
        enabled: yes
    - name: Restart app service
      systemd:
        name: app
        state: restarted
        enabled: yes
  post_tasks:
    - name: Verify service health
      shell: |
        curl -s http://localhost:8080/health | grep -q '"status":"UP"'
      register: health_check
      retries: 5
      delay: 3
      until: health_check.rc == 0
      failed_when: false

The retries, delay, and until keywords ensure that the health check runs up to five times, with a three-second delay between attempts, until the service returns UP. If it fails, the task marks the playbook as failed_when: false, so the overall playbook continues.

Testing and Validation

A playbook that runs in staging but breaks at 3 AM is a broken playbook. Testing must be part of the automation.

Integration Testing with ansible-playbook and --check

Use the --check flag to simulate runs without applying changes. This is crucial for validating new changes before production deployment.

ansible-playbook -i inventory.yml deploy.yml --check --diff --limit staging

The --diff flag shows what would change, and --limit staging targets only the staging environment.

To validate a full deployment, use ansible-playbook with --tags and --skip-tags to isolate parts of the playbook.

ansible-playbook -i inventory.yml deploy.yml --tags=nginx --skip-tags=database

This runs only tasks tagged nginx, skipping database tasks. This is useful for rolling out configuration changes without restarting the entire application.

Validating the Final State

After a playbook runs, verify the system state using assert and shell tasks.

- name: Validate final state
  assert:
    that:
      - "nginx_config.stat.exists"
      - "nginx_config.stat.size > 1024"
      - "app_service.state == 'running'"
    fail_msg: |
      Final state validation failed:
      - Nginx config missing or too small
      - App service not running
    success_msg: "All validation checks passed"

This task fails the entire playbook if any of the conditions are not met. The fail_msg and success_msg are human-readable and appear in the output log.

Continuous Integration and Deployment

A playbook that does not break at 3 AM must be part of a CI/CD pipeline.

Use ansible-lint to enforce coding standards.

- name: Lint Ansible playbooks
  run: |
    ansible-lint --profile=production --fail-on-warning --check-catalog .
  run: |
    ansible-playbook -i inventory.yml test.yml --syntax-check

ansible-lint reads a .ansible-lint file that defines rules like:

---
rules:
  ansible-lint:
    - name: "yaml"
      severity: "warning"
      description: "Check YAML syntax"
    - name: "package"
      severity: "error"
      description: "Ensure all tasks use package module"
  ignore:
    - E303
    - E401

This file ensures that all playbooks follow the project’s standards.

Integrate with git and triggers:

- name: Trigger deployment on new tag
  run: |
    git tag -a v1.2.0 -m "Release v1.2.0"
    git push origin v1.2.0
    ansible-playbook -i inventory.yml deploy.yml --tags=release --limit production

This sequence creates a tag, pushes it, and runs the release tag playbook on production.

Monitoring and Observability

A playbook that does not break at 3 AM must also report its success and failure.

Use debug and log tasks to record key events.

- name: Log deployment start
  debug:
    msg: "Starting deployment of {{ app_name }} (version {{ app_version }})"
    verbosity: 2

- name: Log deployment end
  log:
    message: "Deployment completed successfully"
    facility: "ansible"
    priority: "info"
    tags:
      - "deployment"
      - "success"
      - "app={{ app_name }}"

This produces structured logs that can be collected by rsyslog, fluentd, or journalctl.

For real-time visibility, use command tasks that output JSON to stdout:

- name: Generate deployment report
  command: |
    echo '{'
    echo '  "deployment": {'
    echo '    "host": "{{ inventory_hostname }}",'
    echo '    "started": "{{ ansible_playbook_start_time }}",'
    echo '    "finished": "{{ ansible_playbook_end_time }}",'
    echo '    "status": "success",'
    echo '    "tasks": {{ ansible_playbook_results | length }}'
    echo '  }'
    echo '}'
  register: deployment_report
  run_once: true

The run_once: true ensures that this task runs only once across all hosts, producing a single report per deployment.

Common Pitfalls and Silent Failures

• become: true without become_user: The playbook runs as root, but ansible_user is deploy — tasks are applied by deploy, but become is ignored. Fix: become_user: root.
• notify without handlers: A task notify: Reload nginx, but no handler exists. The task runs, but Reload nginx is never triggered. Use notify: "reload nginx" and define the handler in the handlers section.
• **loop vs with_***: loop is preferred over with_items because it is more predictable and supports map and select filters. Use loop for all iteration tasks.
• register not used: A task runs but its output is not captured. Use register: task_name to capture results for later use.
• delegate_to misapplied: A task runs on the control node, but delegate_to is set to localhost, but the task needs to run on app_server. Use delegate_to: "{{ app_server }}" and verify the inventory file includes app_server as a host.

These patterns, when applied consistently, ensure that Ansible playbooks do not just run — they survive the night.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.