Production engineering guide for automation observability covering patterns, implementation strategies, and operational best practices.
Automation Observability is the practice of instrumenting automated workflows to answer: *What did the automation do, when, and why?* It matters when automation runs unattended—nightly, on-demand, or in response to alerts—and when failures must be diagnosed without human intervention. This is not monitoring of infrastructure or services; it is monitoring of *automation itself*. The same workflow that deploys a service must, in addition to deploying it, log what it did, track its progress, and expose its state and outcomes to engineers, operators, and other automations.
To observe an automation, begin by capturing every execution as a structured event. Each run must include a unique identifier, start and end timestamps, input parameters, and a summary of steps executed. The most common way to achieve this is via a centralized execution log with a standard schema.
A typical automation execution record in JSON form:
{
"execution_id": "e1a3b7c2-d4e5-f6g7-h8i9-j0k1l2m3n4o5",
"workflow": "deploy-service",
"started_at": "2024-01-15T08:30:12Z",
"completed_at": "2024-01-15T08:45:20Z",
"status": "succeeded",
"inputs": {
"service_name": "auth-service",
"version": "v2.3.1",
"env": "production"
},
"steps": [
{
"name": "build-image",
"started_at": "2024-01-15T08:30:15Z",
"completed_at": "2024-01-15T08:35:02Z",
"status": "succeeded",
"output": {
"image_id": "auth-service:v2.3.1",
"layers": 8,
"size_mb": 142
}
},
{
"name": "deploy-to-k8s",
"started_at": "2024-01-15T08:35:05Z",
"completed_at": "2024-01-15T08:44:50Z",
"status": "failed",
"error": {
"type": "k8s-deploy-timeout",
"message": "Deployment rollout timed out after 60s",
"details": [
"Waiting for rollout to finish: 1 of 2 updated replicas are available",
"Pod auth-service-7d4c9b6f58 failed readiness probe: timeout"
]
}
}
]
}The workflow engine must emit this structure to a central store—typically a database, a message queue, or a file system—using a standard interface. A common pattern is to write the execution record to a logs/ directory with the ID as the filename, and to send it to a log aggregation system via curl or jq.
curl -X POST \
-H "Content-Type: application/json" \
-d @- \
https://logs.example.com/api/v1/executions \
< execution.jsonThis record becomes the primary source of truth for understanding what the automation did.
When a step fails, the automation must not only record the failure but also capture all relevant data from that step: logs, outputs, and side effects. A step named apply-terraform might execute:
terraform apply -input=false -auto-approve \
-var-file="env/production.tfvars" \
-out="plan.tfplan"The automation system must capture the output of this command, including terraform plan output and terraform apply result, and include them in the step metadata.
If the step fails, the system must also record the full log of the command:
$ terraform apply -input=false -auto-approve -var-file=env/production.tfvars -out=plan.tfplan
Refreshing state...
...
Plan: 5 to add, 1 to change, 0 to destroy.
...
Apply complete! Resources: 5 added, 1 changed, 0 destroyed.
...
Outputs:
api_url = https://api.production.example.comThe failure reason is not just the exit code, but the full output and any side-effect files. The automation system should automatically archive the plan.tfplan file and embed its path in the step record.
Failure mode: if the plan.tfplan file is not written due to a disk full error, but the step is marked as succeeded, the automation logs appear correct, but the deployment is incomplete. The root cause is silent: the plan was not applied.
Observability is not just about the automation steps, but about linking them to the state of the systems they affect. This requires a common identifier—typically a trace_id—that propagates from the automation execution into the target systems.
When an automation deploys a service to Kubernetes, it sets a trace_id in the deployment manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: auth-service
annotations:
automation.trace_id: e1a3b7c2-d4e5-f6g7-h8i9-j0k1l2m3n4o5
spec:
replicas: 3
selector:
matchLabels:
app: auth-service
template:
metadata:
labels:
app: auth-service
spec:
containers:
- name: app
image: auth-service:v2.3.1The automation system then queries the cluster to verify that the deployment succeeded, using kubectl with --output=json and --watch.
kubectl rollout status deployment/auth-service \
--watch \
--timeout=60s \
--output=json \
--wait \
--output-version=apps/v1The resulting JSON includes the rollout status, current and desired replicas, and conditions. The automation engine merges this data with the execution record, adding a k8s_rollout_status field and a k8s_conditions array.
Failure mode: if the rollout times out but kubectl returns completed status, the automation assumes success. However, the rollout may have stalled at Progressing: true with Ready: false. The automation system must check for status.conditions and evaluate whether Progressing, Available, and ReplicaFailure are all true.
Automations run continuously. Observability must include anomaly detection—automated identification of deviations from expected behavior.
A common pattern is to define a set of *health checks* that run after every automation execution. These checks are simple, idempotent scripts that validate the outcome.
#!/bin/bash
# check-deployment-health.sh
set -euo pipefail
readonly SERVICE_NAME="$1"
readonly TRACE_ID="$2"
readonly EXPECTED_REPLICAS="${3:-3}"
echo "Checking health for $SERVICE_NAME with trace $TRACE_ID"
# Wait for all pods to be ready
kubectl wait --for=condition=ready pod -l app="$SERVICE_NAME" \
--timeout=120s \
--output=json
# Check replica count
kubectl get deployment "$SERVICE_NAME" -o=jsonpath='{.status.replicasReady}' | \
awk -v expected="$EXPECTED_REPLICAS" '$0 == expected {exit 0} {exit 1}'
if [[ $? -ne 0 ]]; then
echo "Health check failed: replica count mismatch"
echo "Expected: $EXPECTED_REPLICAS, Got: $(kubectl get deployment "$SERVICE_NAME" -o=jsonpath='{.status.replicasReady}')"
exit 1
fi
echo "Health check succeeded"
exit 0This script is run as a post-step in the automation workflow:
- name: verify-deployment
run: ./check-deployment-health.sh auth-service e1a3b7c2-d4e5-f6g7-h8i9-j0k1l2m3n4o5 3
depends_on: deploy-to-k8sThe automation system records the result as a new event in the execution log:
{
"type": "health-check",
"name": "verify-deployment",
"started_at": "2024-01-15T08:45:00Z",
"completed_at": "2024-01-15T08:45:15Z",
"status": "succeeded",
"details": {
"output": "Health check succeeded",
"check_type": "replica-count",
"expected": 3,
"actual": 3
}
}Failure mode: if the health check runs but the automation engine does not wait for it, a failure in the check may go unnoticed. The automation marks the deployment as successful, but the health check fails and is logged separately, with no link back to the execution.
Observability includes metrics, not just logs. The automation system must expose and track key metrics—execution duration, success rate, step failure rate, and error types.
Each execution contributes to a time-series database. The following psql commands record metrics for a deploy-service workflow:
-- Insert execution metrics
INSERT INTO automation_metrics (
workflow,
execution_id,
duration_ms,
success,
step_count,
failure_count,
error_types
) VALUES (
'deploy-service',
'e1a3b7c2-d4e5-f6g7-h8i9-j0k1l2m3n4o5',
9118,
true,
4,
1,
ARRAY['k8s-deploy-timeout']
);The system can then run queries to detect anomalies:
-- Find workflows with failure rate above 10% over last 7 days
SELECT
workflow,
COUNT(*) AS total_executions,
SUM(CASE WHEN success THEN 1 ELSE 0 END) AS successes,
(SUM(CASE WHEN success THEN 1 ELSE 0 END) * 100.0 / COUNT(*)) AS success_rate,
COUNT(CASE WHEN NOT success THEN 1 END) AS failures
FROM automation_metrics
WHERE started_at >= NOW() - INTERVAL '7 days'
GROUP BY workflow
HAVING (SUM(CASE WHEN success THEN 1 ELSE 0 END) * 100.0 / COUNT(*)) < 10;Failure mode: if the error_types column is a TEXT[], but the automation engine stores error_types as a JSON object, the query fails to aggregate error types correctly. The ARRAY constructor in INSERT must be aware of the TEXT[] type, and the JSON data must be parsed or flattened.
When an automation fails, the engineer must answer: *Why did it fail?* The answer lies in the full trace of execution, logs, and system state.
The automation system provides a debug endpoint: /api/v1/executions/{execution_id}/debug. This endpoint returns a unified view of:
plan.tfplan, build.log, deployments.yaml)kubectl get all -o yaml, ps aux, docker ps -a)A typical debug response includes a traces array, each trace being a span from the automation engine:
{
"execution_id": "e1a3b7c2-d4e5-f6g7-h8i9-j0k1l2m3n4o5",
"trace_id": "t7b3d9a1-c2e4-f5g6-h7i8-j9k0l1m2n3o4",
"spans": [
{
"name": "build-image",
"start": "2024-01-15T08:30:15Z",
"duration_ms": 2907,
"logs": [
"Building Docker image...",
"Step 1/4: FROM alpine:3.18",
"Step 2/4: COPY app /app",
"Successfully built auth-service:v2.3.1"
],
"artifacts": [
{
"name": "build.log",
"path": "/artifacts/e1a3b7c2-d4e5-f6g7-h8i9-j0k1l2m3n4o5/build.log",
"size_bytes": 4096
}
]
},
{
"name": "deploy-to-k8s",
"start": "2024-01-15T08:35:05Z",
"duration_ms": 5485,
"error": {
"type": "k8s-deploy-timeout",
"message": "Deployment rollout timed out after 60s"
},
"system_state": {
"kubectl": {
"get": {
"deployments": [
{
"name": "auth-service",
"status": {
"ready_replicas": 2,
"replicas": 3,
"conditions": [
{
"type": "Progressing",
"status": "True",
"reason": "NewReplicaSetAvailable"
},
{
"type": "Available",
"status": "False",
"reason": "MinimumReplicasUnavailable"
}
]
}
}
]
}
}
}
}
]
}The automation system exposes this as a REST endpoint, and engineers can fetch it directly:
curl https://automation.example.com/api/v1/executions/e1a3b7c2-d4e5-f6g7-h8i9-j0k1l2m3n4o5/debugFailure mode: if the system_state is not captured at the time of failure, but only after the step completes, the state may not reflect the exact moment the rollout failed. The automation must capture state immediately upon step failure, not after kubectl returns.
The most common failure is assuming that a successful automation run is a correct run. But a run that completes without error is not necessarily correct. Observability reveals that automation is not just execution, but *execution with understanding*.
This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy