Compliance as Code: Automating Audit Evidence Collection

Turn regulatory compliance from a manual documentation exercise into an automated engineering practice. Covers policy-as-code, automated evidence collection, continuous compliance monitoring, audit preparation workflows, and the infrastructure that makes auditors happy and engineers productive.

Compliance as Code (CaC) treats audit evidence not as static artifacts but as first-class, versioned, and automated artifacts within the software lifecycle. It matters when auditors demand proof of controls—such as access reviews, data retention policies, or encryption configurations—and when teams must reprove compliance across multiple frameworks (ISO 27001, HIPAA, SOC 2) with minimal manual effort. CaC is essential during audit cycles, regulatory changes, or system migrations, where the cost of gathering evidence by hand exceeds the cost of maintaining its automation.

Defining the Audit Evidence Artifact

Audit evidence in CaC is a structured, executable artifact that documents *how* and *when* a control was validated. Each evidence item is a self-contained unit, typically a file in a known location, with a canonical structure:

{
  "evidence_id": "access-review-2024-q1",
  "control_id": "A.12.3",
  "framework": "ISO 27001",
  "evidence_type": "access-review",
  "timestamp": "2024-03-15T12:34:56Z",
  "metadata": {
    "validated_by": "user:[email protected]",
    "validated_at": "2024-03-15T12:34:56Z",
    "version": "1.2.0",
    "source": "https://github.com/org/compliance/commit/abc123"
  },
  "results": {
    "total_users": 127,
    "reviewed_users": 127,
    "unreviewed_users": 0,
    "overdue_reviews": 5,
    "report_url": "https://access-reviews.example.com/reports/2024-q1"
  }
}

This JSON file is generated by a script or pipeline, stored in a version-controlled directory (/compliance/evidence/), and indexed in a centralized audit log. The evidence ID is unique and deterministic, derived from a hash of the control, date, and configuration.

Generating Evidence from Control Configurations

Each compliance control is defined as a configuration file, written in YAML, that specifies *what* to check, *how* to collect evidence, and *when* to run it.


control_id: A.12.3
framework: ISO 27001
title: "Periodic Access Review"
description: "Ensure all users have reviewed their access rights quarterly."
frequency: "quarterly"
schedule: "0 0 1 1,4,7,10 *"
run:
  - command: "python3 -m compliance.access_review"
    args:
      - "--days-since-last-review=90"
      - "--include-deactivated=true"
      - "--output=/compliance/evidence/access-review-2024-q1.json"
    environment:
      DATABASE_URL: "postgres://prod:5432/iam"
      REPORTING_API_KEY: "sk-abc123"
    timeout: "1h"
  - command: "curl"
    args:
      - "-X"
      - "POST"
      - "https://audit-log.example.com/evidence"
      - "-H"
      - "Content-Type: application/json"
      - "-d"
      - "@/compliance/evidence/access-review-2024-q1.json"
    depends_on: ["access_review"]

The run section defines a sequence of steps. Each step may be a shell command, a script, or a service call. The depends_on field ensures that evidence is only submitted after all preconditions are met.

Automating Evidence Collection with CI/CD

Evidence collection is triggered by a CI/CD pipeline. A Git commit to controls/ directory automatically schedules a new run.

# .github/workflows/compliance.yml
name: Compliance Automation
on:
  push:
    paths:
      - "controls/**"
      - ".compliance/config.yaml"
  schedule:
    - cron: "0 0 * * 1"  # Every Monday at midnight

jobs:
  generate-evidence:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt
      - run: |
          python3 -m compliance.discover_controls
          for control in $(find controls -name "*.yaml"); do
            python3 -m compliance.run_control "$control"
          done
      - name: Upload Evidence
        uses: actions/upload-artifact@v3
        with:
          name: compliance-evidence
          path: /compliance/evidence/
          if-no-files-found: "ignore"
      - name: Publish to Audit Log
        run: |
          for file in /compliance/evidence/*.json; do
            curl -X POST \
              -H "Authorization: Bearer $AUDIT_LOG_TOKEN" \
              -H "Content-Type: application/json" \
              -d "@$file" \
              https://audit-log.example.com/evidence
          done

The pipeline runs every time a control changes or on a fixed schedule. It discovers all controls, executes them in parallel, and uploads the resulting evidence files.

Validating Evidence with Schema and Checks

Before evidence is accepted, it must pass schema validation and control-specific checks.

// schema/evidence.schema.json
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["evidence_id", "control_id", "framework", "timestamp"],
  "properties": {
    "evidence_id": { "type": "string", "pattern": "^[a-z0-9-]+$" },
    "control_id": { "type": "string", "pattern": "^\\w+\\.\\d+$" },
    "framework": { "type": "string", "enum": ["ISO 27001", "HIPAA", "SOC 2", "GDPR"] },
    "results": { "type": "object", "required": ["total_users", "reviewed_users"] },
    "metadata": {
      "type": "object",
      "properties": {
        "validated_by": { "type": "string", "format": "email" },
        "version": { "type": "string", "pattern": "^\\d+\\.\\d+\\.\\d+$" }
      }
    }
  },
  "additionalProperties": false
}

The pipeline includes a validation step:

# validate-evidence.sh
#!/bin/bash
set -euo pipefail

SCHEMA_PATH="/schema/evidence.schema.json"

for evidence in /compliance/evidence/*.json; do
  if ! jq -e --argjson schema "$(cat "$SCHEMA_PATH")" \
    'def validate: . as $in | $schema | . | (inputs | .properties | reduce . as $prop ({}; . + {($prop.key): $prop.value})) | validate($in)' \
    "$evidence"; then
    echo "ERROR: Evidence $evidence failed schema validation"
    exit 1
  fi
done

Failure to validate results in an immediate pipeline failure, with a clear error message:

ERROR: Evidence access-review-2024-q1.json failed schema validation
{
  "message": "Instance failed to validate against schema",
  "errors": [
    {
      "keyword": "required",
      "message": "Missing required property: results.overdue_reviews",
      "params": { "missing": "overdue_reviews" },
      "schema": { "path": "/properties/results" },
      "instance": { "path": "/results" }
    }
  ]
}

This error indicates that the access review script did not emit the overdue_reviews field, even though it was expected.

Managing Evidence Provenance and Traceability

Evidence is tied to the exact configuration state that generated it. The pipeline records provenance in a provenance.json file included with each evidence artifact.

{
  "source": "https://github.com/org/compliance/commit/abc123",
  "pipeline_id": "compliance-20240315-123456",
  "environment": {
    "CI": true,
    "GITHUB_REPOSITORY": "org/compliance",
    "GITHUB_REF": "refs/heads/main",
    "GITHUB_SHA": "abc123def456",
    "GITHUB_RUN_ID": 123456,
    "GIT_COMMIT": "abc123def4567890123456789012345678901234"
  },
  "execution": {
    "started_at": "2024-03-15T12:34:56Z",
    "finished_at": "2024-03-15T13:02:11Z",
    "duration": "27m 15s",
    "status": "success",
    "logs_url": "https://ci.example.com/runs/123456"
  }
}

This provenance is used to answer auditor questions like "How was this evidence generated?" or "What changed between the last and current run?" It’s stored alongside the evidence and served via a /evidence/{id}/provenance endpoint.

Handling Evidence Lifecycle and Retention

Evidence is not static. It must be versioned, archived, and retained according to policy.

# rotate-evidence.sh
#!/bin/bash
set -euo pipefail

EVIDENCE_DIR="/compliance/evidence"
ARCHIVE_DIR="/compliance/archive"

# Move evidence older than 1 year to archive
find "$EVIDENCE_DIR" -name "*.json" -mtime +365 -exec mv {} "$ARCHIVE_DIR" \;

# Archive evidence with metadata
for file in "$ARCHIVE_DIR"/*.json; do
  evidence_id=$(jq -r '.evidence_id' "$file")
  mv "$file" "$ARCHIVE_DIR/$evidence_id.json"
  gzip -c "$ARCHIVE_DIR/$evidence_id.json" > "$ARCHIVE_DIR/$evidence_id.json.gz"
done

# Update index
echo "Last rotated: $(date -Iseconds)" > "$ARCHIVE_DIR/.last-rotated"

Retention is enforced by a nightly cron job:

# /etc/cron.daily/compliance-rotation
0 2 * * * /opt/compliance/bin/rotate-evidence.sh

Evidence older than 10 years is moved to cold storage (e.g., S3, Glacier), with a cold_storage flag in the metadata.

Failure Modes and Common Pitfalls

The most common failure is evidence being generated but not submitted to the audit log. This occurs when the curl step in the pipeline fails due to a network timeout, missing API key, or incorrect auth header.

curl: (7) Failed to connect to audit-log.example.com port 443: Connection timed out

This error is silent: the evidence file exists, but the audit log does not know it. A daily monitoring script checks for missing evidence:

# check-missing-evidence.sh
#!/bin/bash
set -euo pipefail

LAST_RUN=$(date -d "yesterday" -I)
EVIDENCE_IDS=$(curl -s https://audit-log.example.com/evidence?since="$LAST_RUN" | jq -r '.evidence_ids[]')

while IFS= read -r id; do
  if [ ! -f "/compliance/evidence/$id.json" ]; then
    echo "WARNING: Evidence $id not found in local storage"
    # Send alert via Slack or email
  fi
done <<< "$EVIDENCE_IDS"

Another failure arises from incorrect control_id format. The framework expects A.12.3, but a team writes A12.3 or A.12.31. This leads to missing evidence in reports, especially when the audit log queries by control_id.

A subtle issue is the schedule field in control configuration. The cron format is used, but the pipeline runs on a different timezone than the audit system. The schedule field must include timezone information:

schedule: "0 0 1 1,4,7,10 *"
timezone: "UTC"

Without explicit timezone, quarterly reviews run on the 1st of each quarter but in the wrong time zone, leading to off-cycle audits.

Reviewing Evidence in Production

Auditors inspect evidence by querying the audit log. A typical query returns a list of evidence items, with metadata and results:

GET /evidence?control_id=A.12.3&framework=ISO%2027001&from=2024-01-01&to=2024-12-31 HTTP/1.1
Host: audit-log.example.com
Accept: application/json
Authorization: Bearer sk-abc123

Response:

[
  {
    "evidence_id": "access-review-2024-q1",
    "control_id": "A.12.3",
    "framework": "ISO 27001",
    "timestamp": "2024-03-15T12:34:56Z",
    "results": {
      "total_users": 127,
      "reviewed_users": 127,
      "overdue_reviews": 5
    },
    "provenance_url": "https://ci.example.com/runs/123456"
  }
]

Auditors can drill down into each evidence item, view its provenance, and verify that the evidence matches the control definition. The system supports versioning: GET /evidence/access-review-2024-q1?version=1.2.0 returns the exact state of the evidence at that version.

Summary

Compliance as Code turns audit evidence into a first-class, executable, and versioned artifact. It automates the collection of evidence from configuration-driven controls, validates it against schemas, and ties it to provenance. The system is resilient to failure modes such as missed submissions, incorrect scheduling, and schema mismatches. By standardizing the structure, generation, validation, and lifecycle of evidence, CaC enables auditors to trust evidence without manual review. The process is repeatable, traceable, and self-documenting.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.