Compliance Monitoring | The Garnet Wiki

Compliance monitoring is the continuous process of observing system configuration, runtime state, and data flows to confirm they remain within defined regulatory, contractual, or internal policy boundaries. It matters when you must produce evidence for auditors, detect drift before it becomes a breach, or enforce separation‑of‑duties across ephemeral infrastructure. Unlike a point‑in‑time audit, monitoring runs on a schedule or in response to events, generating alerts, evidence artifacts, and, optionally, automated remediation actions.

Verify the Current State of Controls

Before monitoring can surface problems, you need a live snapshot of what is actually configured. This step queries the running system and presents the raw state so it can be compared against policy definitions.

A concrete query using the standard Kubernetes CLI:

kubectl get clusterrolebinding --all-namespaces -o json | jq '
  .items[] | select(
    .metadata.labels."compliance" == "required"
  ) | {
      name: .metadata.name,
      namespace: .metadata.namespace,
      subjects: .subjects | length
    }
'

kubectl get with -o json streams the full resource list; jq filters for bindings that carry a compliance: required label and condenses the output to the fields an operator typically needs. The --all-namespaces flag ensures every namespace is inspected without repeated invocations. If your cluster uses a different enforcement mechanism, the same pipeline applies: kubectl get <resource> -o json | jq '<filter>'.

An equivalent check against a PostgreSQL‑backed policy store uses psql:

psql -U compliance_reader -d audit -c \
  "SELECT policy_id, resource_type, resource_id, current_state, last_checked_at \
   FROM policy_snapshot \
   WHERE tenant_id = current_setting('app.tenant_id') \
   ORDER BY resource_type, resource_id;"

psql connects to the audit database, and the query returns the most recent snapshot row per resource, ordered so the latest entry per resource appears first. Adjust the WHERE clause to match your tenant‑identification pattern (environment variable, config file, or connection‑parameter).

Detect Drift and Unauthorized Changes

Drift detection compares the live state you just captured against the desired state stored in your policy repository. The goal is to surface any resource that has moved away from the approved configuration without a corresponding change request.

A drift‑detection flow that combines Git with the Kubernetes API:

git diff HEAD -- "compliance/policies/*.yaml" | \
kubectl apply --dry-run=client -f - 2>&1 | \
grep -E "^error|^Warning:"

git diff produces the uncommitted changes in your policy‑as‑code repository. The pipe sends those changes to kubectl apply in dry‑run mode, which validates the manifests against the cluster’s schema without persisting them. The grep step surfaces any error messages or warnings that would prevent the desired state from being realized. If the output is clean, the live cluster can be reconciled with the repository version; any output indicates either a syntax issue in the policy or a drift that must be resolved before monitoring can consider the state healthy.

On bare‑metal or VM fleets, the same logic reduces to a file‑integrity check:

find /etc/security /var/lib/app/config -type f -name "*.conf" -exec md5sum {} \; | sort > /tmp/current-checksums.txt
diff /etc/compliance/baseline-checksums.txt /tmp/current-checksums.txt

The diff output tells you exactly which files changed checksums, which you then investigate against your change‑management process.

Produce Audit-Ready Evidence

Auditors typically want a single, immutable record that shows what was checked, when, and by whom. The standard mechanics are: run a deterministic query, capture the output, compute a cryptographic hash, and store the hash in an append‑only log or object store.

Evidence generation using psql and sha256sum:

psql -U auditor -d audit -c \
  "COPY (SELECT policy_id, resource_id, current_state, checked_at \
       FROM policy_snapshot \
       WHERE checked_at >= now() - interval '24 hours') \
   TO STDOUT WITH (FORMAT csv, HEADER true)" | \
sha256sum > /var/evidence/audit-$(date +%Y%m%d-%H%M%S).hash

COPY … TO STDOUT streams the query result as CSV, and sha256sum produces a hash that can be deposited in an immutable storage bucket. The hash is the piece an auditor checks: if the same query run tomorrow produces a different hash, you have evidence of a change in the window.

For HTTP‑exposed services, a comparable pattern uses curl and jq:

curl -s "$COMPLIANCE_API/policies?since=yesterday" | \
jq -S . > /tmp/policies-yesterday.json && \
sha256sum /tmp/policies-yesterday.json > /var/evidence/policy-hash.json

The pipeline—fetch, normalize, hash—is what matters for evidence integrity. $COMPLIANCE_API and any auth headers are organization‑specific; the surrounding pattern is portable.

Configure Alerting and Escalation

When monitoring detects a policy violation, the system must produce a notification that reaches the right people quickly. The pipeline is: monitoring tool emits an event → alert routing rules → escalation matrix → integration with your incident‑management platform.

A real‑world alert using systemd and journalctl:

systemctl status auditd | grep -q 'FAIL' && \
journalctl -u auditd -n 1 --no-pager | \
curl -s -X POST -H "Content-type: application/json" \
     -d "$(journalctl -u auditd -n 1 --output=json | jq '.message')" \
     "$NOTIFICATION_ENDPOINT"

systemctl status auditd checks whether the audit daemon reports a failure. journalctl -u auditd -n 1 --no-pager extracts the most recent log line. The curl command posts that message to a notification endpoint. The systemctl and journalctl flags are real; $NOTIFICATION_ENDPOINT is your organization’s integration point (could be a webhook, email gateway, or ticketing system).

If you use a dedicated alerting engine, the equivalent is a rule expression (Prometheus, Thanos, etc.) that evaluates to true when a compliance metric crosses a threshold, then forwards to your alertmanager configuration. The mechanics—metric evaluation, routing tree, receiver map—are the same regardless of the product.

Integrate with Incident Response

When a compliance breach is confirmed, the response workflow typically triages the severity, preserves the current state for later analysis, and initiates remediation or ticket creation.

A concrete IR‑trigger using kubectl and curl:

VIOLATION=$(kubectl get constrainttemplate my-policy -o json | \
            jq -r '.spec.metadata.description')
if [ -n "$VIOLATION" ]; then
  curl -s -X POST -H "Content-type: application/json" \
       -d "{\"title\":\"Compliance violation: $VIOLATION\", \
            \"severity\":\"high\", \
            \"source\":\"compliance-monitor\", \
            \"details\":$(kubectl get constrainttemplate my-policy -o json)}" \
       "$INCIDENT_CREATION_API"
fi

kubectl get constrainttemplate my-policy -o json fetches the policy definition, and jq -r extracts its description. The if block constructs a JSON payload and posts it to an incident‑creation API. The kubectl get and jq commands are real; $INCIDENT_CREATION_API is your service‑desk integration endpoint.

In a server‑less or agent‑based setup, the same flow often uses a webhook from your monitoring platform to trigger a CI/CD pipeline step that either auto‑remediates (if the drift is low‑risk) or creates a ticket with the captured state snapshot.

Sharp Edges

What silently fails, which options are confused for each other, and what you only learn after it breaks in production.

Silent drift between policy‑as‑code repos and live enforcement: Many teams store policies in Git but never sync them to the enforcement point. The monitoring page shows the live state, but the desired state is stale, so every check appears healthy until an auditor asks a question you can’t answer. Fix it by automating a CI job that pushes the latest policy bundle to every enforcement point after every merge.
Confusing “last_checked_at” with “last_compliant_at”: A timestamp column that records when a query ran does not mean the resource was compliant at that moment. It only means the check executed. If your evidence relies on that column, you’ll report compliance windows that include periods of known violation. Add a compliant_until or violation_detected_at field to your snapshot schema.
jq pipe failures when output is empty: kubectl get ... | jq '...' returns null and exits 0 when no resources match, which can hide a real problem (e.g., all NetworkPolicies were deleted accidentally). Always pair the pipe with a conditional: kubectl get ... | jq '...' // error "no matching resources found" or check the exit code with kubectl get ... || echo "kubectl failed".
Audit‑hash collisions in high‑throughput environments: When you stream thousands of policy snapshots per minute into a single hash file, the hash becomes a rolling window rather than a point‑in-time snapshot. Auditors may reject it as non‑deterministic. Use per‑resource hashes and a manifest file that lists each hash with its timestamp instead of a single aggregate value.
systemctl service name mismatches: On some distributions, the audit daemon may be named auditd, on others audit or auditd.service. Relying on a hard‑coded service name in scripts causes false‑negative alerts on newly provisioned nodes. Prefer systemctl list-units --type=service | grep audit or query your configuration management database for the canonical service name.
Tenant‑id leakage in multi‑tenant psql queries: If your compliance snapshot table doesn’t row‑level‑security, a query that filters by tenant_id may accidentally return rows from other tenants if the column is null or mis‑indexed. Always wrap tenant‑filtered queries with AND tenant_id IS NOT NULL and, if possible, enable row‑level security on the compliance schema.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.