Monitoring as Code: Version-Controlled Dashboards and Alerts

Monitoring as Code is the practice of storing monitoring configuration—dashboards, alerts, and metric definitions—in version-controlled text files rather than in UIs or proprietary binary formats. This approach makes monitoring infrastructure reproducible, auditable, and deployable like application code. It matters when teams need to maintain consistent alerting across environments, automate monitoring setup, or ensure that monitoring changes follow the same review and deployment process as application code.

Creating and Managing Alert Definitions

Alert definitions are typically stored as structured text files in a repository. The most common format is YAML, but JSON is also used. An alert definition includes the condition that triggers the alert, the severity level, and metadata for routing and notifications.

Basic Alert Definition Format

name: "High CPU Usage"
description: "Instance CPU usage exceeds 80% for 5 minutes"
condition:
  metric: cpu_usage
  operator: ">="
  threshold: 80
  duration: 300
severity: "warning"
labels:
  team: "backend"
  service: "webapp"

Alert Conditions and Operators

Alert conditions require a metric name, an operator, and a threshold. The supported operators vary by system but typically include:

• > Greater than
• < Less than
• >= Greater than or equal
• <= Less than or equal
• == Equal to
• != Not equal to

The duration field specifies how long the condition must persist before triggering an alert, measured in seconds.

Alert Routing and Notifications

Alerts can be routed based on labels. A system might define routes that send different alerts to different channels:

name: "Critical Database Alert"
condition:
  metric: database_errors
  operator: ">="
  threshold: 10
  duration: 60
labels:
  team: "database"
  severity: "critical"

Example Alert Management Script

To manage alerts in a version-controlled manner, you can write a script that applies definitions from files to a monitoring system:

#!/bin/bash
# Apply all alert definitions from the alerts directory
for file in alerts/*.yaml; do
  curl -X POST \
    -H "Content-Type: application/yaml" \
    -d "@$file" \
    http://monitoring-api.example.com/v1/alerts
done

Common Alert Configuration Issues

A common failure mode occurs when alert conditions reference metrics that don't exist in the backend. The system may silently ignore alerts with invalid metric names, leading to silent monitoring gaps.

Creating and Managing Dashboards

Dashboards are defined as structured configuration files that specify which metrics to display, how to visualize them, and what data ranges to show.

Dashboard Definition Structure

title: "Application Performance Dashboard"
description: "Overview of key application metrics"
widgets:
  - type: "graph"
    title: "CPU Usage"
    metric: cpu_usage
    range: "last_1h"
    aggregation: "avg"
  - type: "table"
    title: "Error Rates"
    metrics:
      - error_rate_5xx
      - error_rate_4xx
    range: "last_15m"

Widget Types and Configuration

Widgets can be graphs, tables, or single-value displays. Each widget specifies:

• The metric(s) to display
• The visualization type (graph, table, etc.)
• The time range for data
• Aggregation method (average, sum, count)

Dashboard Versioning Considerations

When modifying dashboards, always version the configuration files. A dashboard file might be named application-dashboard-v1.yaml and application-dashboard-v2.yaml to track changes.

Example Dashboard Deployment Script

#!/bin/bash
# Deploy dashboard from YAML definition
curl -X PUT \
  -H "Content-Type: application/yaml" \
  -d @dashboards/application-dashboard.yaml \
  http://monitoring-api.example.com/v1/dashboards/application

Integrating with Version Control

All monitoring definitions should be stored in the same repository as application code. This ensures that changes to monitoring align with code changes and are subject to the same review process.

Git Hooks for Alert Validation

A pre-commit hook can validate alert definitions before allowing commits:

#!/bin/bash
# Pre-commit hook to validate alert files
for file in alerts/*.yaml; do
  if ! yamllint "$file"; then
    echo "Invalid YAML in $file"
    exit 1
  fi
done

Branch Protection Rules

Use branch protection rules to require review before merging monitoring changes. This prevents accidental deployment of broken or incorrect alert configurations.

Managing Alert Dependencies and Cascading Failures

Alerts often depend on other alerts or metrics. A failure in one metric can cause a cascade of alerts.

Example: Alert Dependency Chain

# Primary alert
name: "Database Connection Pool Exhausted"
condition:
  metric: database_pool_exhausted
  operator: "=="
  threshold: 1
  duration: 60

# Secondary alert that depends on primary
name: "Application Down"
condition:
  metric: app_status
  operator: "=="
  threshold: "down"
  duration: 300
labels:
  depends_on: "Database Connection Pool Exhausted"

Alert Suppression and Silence Rules

Some systems support suppression rules that prevent alerts from firing during known maintenance windows:

name: "Maintenance Window Suppression"
condition:
  metric: maintenance_window_active
  operator: "=="
  threshold: 1
  duration: 3600

Handling Alert State and History

Monitoring systems track alert state including when alerts were triggered, acknowledged, or resolved. This information should be preserved and versioned alongside alert definitions.

Alert State Persistence

Alerts are typically persisted in a database with fields like:

• alert_id: Unique identifier for the alert
• status: Active, acknowledged, resolved
• timestamp: When state changed
• acknowledged_by: User who acknowledged

Deployment and Rollback Strategies

Monitoring changes should be deployed with the same care as application code. Changes can be rolled back if they cause issues.

Deployment Rollback Example

# Deploy new alert configuration
curl -X POST \
  -H "Content-Type: application/yaml" \
  -d @alerts/new-alerts.yaml \
  http://monitoring-api.example.com/v1/alerts

# If issues arise, rollback to previous version
curl -X DELETE \
  http://monitoring-api.example.com/v1/alerts/new-alerts

curl -X POST \
  -H "Content-Type: application/yaml" \
  -d @alerts/old-alerts.yaml \
  http://monitoring-api.example.com/v1/alerts

Common Pitfalls and Silent Failures

Metric Name Mismatches

One of the most common silent failures is when alert definitions reference metrics that don't exist in the monitoring backend. The system may silently ignore these alerts, leaving teams unaware of missing monitoring.

Invalid Time Ranges

Time range specifications in dashboards can silently default to incorrect values if an unsupported range is specified. For example, specifying "last_30d" when only "last_1h", "last_6h", "last_1d" are supported.

Label Value Validation

Labels used for routing alerts may be silently truncated or ignored if they exceed system limits. A label value of 256 characters might be silently truncated to 128 characters.

Alert Lifecycle Management

Alerts should have a defined lifecycle from creation to retirement. This includes:

• Creation with clear ownership
• Periodic review for relevance
• Retirement when no longer needed

Example: Alert Lifecycle Script

#!/bin/bash
# List all active alerts that haven't been reviewed in 90 days
curl http://monitoring-api.example.com/v1/alerts \
  | jq '.[] | select(.last_reviewed < "2023-01-01")'

Testing Alert Definitions

Alert definitions should be tested before deployment. This can be done by:

• Validating YAML syntax
• Simulating alert conditions
• Checking metric availability

Example Test Script

#!/bin/bash
# Validate alert files and check for missing metrics
for file in alerts/*.yaml; do
  echo "Testing $file"
  yamllint "$file"
  # Extract metrics from alert definition
  metrics=$(grep -o 'metric: [a-zA-Z0-9_]*' "$file" | cut -d' ' -f2)
  for metric in $metrics; do
    if ! curl -f -s "http://monitoring-api.example.com/v1/metrics/$metric" > /dev/null; then
      echo "Warning: Metric $metric not found"
    fi
  done
done

Alert Ownership and Metadata

Alert definitions should include metadata about ownership, contact information, and documentation:

name: "High Memory Usage"
description: "Instance memory usage exceeds 85% for 10 minutes"
owner: "backend-team"
contact: "[email protected]"
documentation: "https://internal.company.com/docs/alerts/high-memory-usage"
condition:
  metric: memory_usage
  operator: ">="
  threshold: 85
  duration: 600

Handling Multiple Environments

Monitoring configurations often vary between environments (dev, staging, production). Version-controlled configurations can support this through environment-specific files or templating.

Example Environment-Specific Alert File

# alerts/prod-alerts.yaml
name: "Production Critical Alert"
condition:
  metric: critical_errors
  operator: ">="
  threshold: 5
  duration: 300
labels:
  environment: "production"

This approach ensures that monitoring configuration changes are tracked and deployed consistently across environments, maintaining the same monitoring coverage and alerting behavior.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.