SLO-Based Alerting

Replace threshold-based alerts with SLO-driven alerting that reduces noise and focuses on user impact. Covers error budgets, burn rate alerts, multi-window strategies, alert routing, and the patterns that eliminate alert fatigue while catching real incidents.

SLO-based alerting triggers notifications when service reliability metrics fall below defined thresholds. It is used during incidents to identify when performance degradation exceeds acceptable levels, and in monitoring to proactively detect system instability before it impacts users.

Configure Alert Rules Based on Error Budget Consumption

Alert rules should trigger when error budget consumption reaches a defined percentage. For example, if a service has a 99.9% availability SLO with a 1% error budget, an alert might fire when 0.5% of the error budget is consumed.

To define such rules in Prometheus, use recording rules and alerting rules:

# Recording rule to calculate error budget consumption
- record: service_error_budget_consumed
  expr: 1 - (rate(http_requests_total{status=~"2.."}[5m]) / rate(http_requests_total[5m]))

# Alert rule that fires when error budget consumption exceeds threshold
- alert: HighErrorBudgetConsumption
  expr: service_error_budget_consumed > 0.005
  for: 5m
  labels:
    severity: page
  annotations:
    summary: "High error budget consumption"

The for duration ensures the alert only fires after sustained degradation, not transient spikes. A common mistake is setting the duration too short (e.g., for: 1m), which can result in noisy alerts for temporary issues.

Set Alert Thresholds Using Percentage of Error Budget

Alert thresholds must be based on remaining error budget rather than absolute values. For instance, a service with a 99.5% SLO has 0.5% error budget. If the system consumes 0.3% of this budget over five minutes, it is at 60% of its error budget.

Use Prometheus expressions to compute error budget consumption:

# Calculate remaining error budget percentage
- record: service_error_budget_remaining
  expr: 1 - service_error_budget_consumed

Alerts should be defined using this metric with a threshold that aligns with the SLO's acceptable risk level. The typical configuration includes:

Define Alert Severity Based on Error Budget Consumption Rate

Alert severity should correlate with error budget consumption rate. For example, a service consuming 0.1% of its error budget in one minute might trigger a warning, while 0.3% in one minute could trigger a page.

In Prometheus, this is expressed by defining multiple alert rules with different thresholds:

# Warning level alert
- alert: LowErrorBudgetConsumption
  expr: service_error_budget_consumed > 0.001
  for: 1m
  labels:
    severity: warning

# Page level alert
- alert: HighErrorBudgetConsumption
  expr: service_error_budget_consumed > 0.003
  for: 1m
  labels:
    severity: page

The for duration must be consistent across alert levels to avoid overlapping alerts and confusion.

Alerts from multiple services sharing a common SLO should be grouped together. This helps incident responders quickly assess the scope of an issue affecting several components.

In Prometheus, use labels to group alerts:

# Alert rule with grouping label
- alert: ServiceDegradation
  expr: service_error_budget_consumed > 0.002
  for: 5m
  labels:
    severity: page
    team: backend

Use the Prometheus alertmanager configuration to define groupings:

route:
  group_by: [team, alertname]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h

This ensures related alerts are sent together in a single notification. The group_wait is critical to prevent premature alert grouping.

Configure Alert Suppression for Known Issues

Alerts should be suppressed when known issues or maintenance periods are active. This avoids unnecessary noise during planned downtimes or known degraded conditions.

In Prometheus, use silence rules:

# Create a silence in the alertmanager API
curl -X POST http://alertmanager.example.com/api/v1/silences \
  -H "Content-Type: application/json" \
  -d '{
    "matchers": [
      {"name": "job", "value": "backend-service"},
      {"name": "alertname", "value": "HighErrorBudgetConsumption"}
    ],
    "startsAt": "2024-12-01T10:00:00Z",
    "endsAt": "2024-12-01T12:00:00Z",
    "createdBy": "sre-team",
    "comment": "Scheduled maintenance"
  }'

Silences are defined by matchers and a time window. The createdBy and comment fields help with auditability.

Set Alert Thresholds for Latency SLOs

For latency-based SLOs, use quantile-based metrics to define alert thresholds. If a service promises 95th percentile response time under 200ms, an alert should trigger when this exceeds 300ms for sustained periods.

Prometheus example:

# Record latency quantile
- record: service_latency_p95
  expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Alert on excessive latency
- alert: HighLatency
  expr: service_latency_p95 > 0.3
  for: 5m
  labels:
    severity: page

The histogram_quantile function requires a histogram metric with appropriate buckets. Misconfiguring bucket sizes can lead to false alerts or missed degradation.

Implement Alert Escalation Based on Time Spent in Error State

Alerts should escalate based on how long they remain in error state. A 5-minute alert that has been active for 10 minutes should trigger escalation.

Prometheus does not support automatic escalation natively. Instead, use a combination of alert rules and external alerting logic:

# Alert with multiple time windows
- alert: LongRunningError
  expr: service_error_budget_consumed > 0.002
  for: 10m
  labels:
    severity: escalation

This requires manual configuration or integration with a system that supports time-based escalation rules.

Define Alert Suppression Based on Time Windows

Alerts should be suppressed during known maintenance windows or when specific conditions are met. This prevents alerting during planned operations.

In Prometheus, use the inhibit feature to suppress alerts:

# Inhibit rule
- inhibit_rule:
  source_matchers:
    - name: alertname
      value: "MaintenanceInProgress"
  target_matchers:
    - name: severity
      value: "page"

This prevents page-level alerts when a maintenance alert is active.

Monitor Alert Effectiveness with Alerting Metrics

Track how often alerts fire and how many are resolved. This helps validate the accuracy of alert thresholds.

In Prometheus, record metrics for alert activity:

# Record alert firing rate
- record: alerts_fired_total
  expr: increase(alerts_fired[1h])

# Record alert resolution time
- record: alert_resolution_time_seconds
  expr: avg_over_time(alerts_resolved[1h])

These metrics help tune thresholds and reduce false positives.

Handle Alert Noise Through Rate Limiting

Alert noise can overwhelm incident responders. Implement rate limiting to control alert frequency.

In Alertmanager, use the rate_limit feature:

# Route with rate limiting
route:
  group_by: [alertname]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: "pagerduty"
  routes:
    - match:
        severity: page
      receiver: "pagerduty"
      repeat_interval: 30m

The repeat_interval prevents repeated notifications for the same alert. Setting it too long can delay resolution.

Manage Alert Dependencies with Dependency Tracking

Some alerts depend on other services or infrastructure. These dependencies must be tracked and considered in alerting logic.

Example of a dependency-based alert:

# Alert that depends on upstream service
- alert: DependentServiceDegradation
  expr: |
    (service_error_budget_consumed > 0.002)
    AND
    (upstream_service_available == 0)
  for: 5m
  labels:
    severity: page

This requires tracking the availability of dependent services, which may involve external monitoring.

Troubleshoot Alert Delays and Latency

Alerts may not fire immediately due to Prometheus scrape intervals or alertmanager processing delays. A 5-minute alert may not trigger for up to 10 minutes after degradation begins.

To reduce alert delay:

If alerts are delayed, check Prometheus metrics like prometheus_rule_evaluation_duration_seconds and alertmanager_alerts_sent_total.

Handle Alert False Positives with Additional Conditions

Alerts should include additional conditions to reduce false positives. For example, an alert for high error budget consumption should also check that the service is not under load.

Example:

# More precise alert rule
- alert: HighErrorBudgetConsumptionWithLoad
  expr: |
    (service_error_budget_consumed > 0.002)
    AND
    (rate(http_requests_total[1m]) > 100)
  for: 5m
  labels:
    severity: page

This ensures alerts fire only when both degradation and load are present.

Validate Alert Logic with Synthetic Tests

Use synthetic traffic to validate alert logic. This includes injecting known issues into the system and verifying that alerts fire correctly.

Example command to generate synthetic load:

# Generate load for testing
ab -n 1000 -c 10 http://service.example.com/api/endpoint

Monitor Prometheus metrics during the test to confirm alert behavior.

Set Alert Thresholds Based on Historical Data

Use historical data to set alert thresholds. For example, if a service typically experiences 0.05% error budget consumption during normal operation, a threshold of 0.1% might be appropriate.

Query historical data:

# Query error budget consumption over past week
prometheus-cli query 'rate(http_requests_total{status!~"2.."}[5m]) / rate(http_requests_total[5m])'

Use this data to tune thresholds and reduce false positives.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.