SLOs That Drive Real Reliability: From Error Budgets to Engineering Decisions

Implement Service Level Objectives that actually improve reliability. Covers SLI selection, error budget policies, burn rate alerting, and the organizational negotiations that make SLOs work in practice.

This page describes how to define, manage, and use Service Level Objectives (SLOs) in a way that directly impacts engineering decisions and system reliability. It focuses on the practical mechanics of error budgeting, including how to calculate budgets, set alerting rules, and translate SLO violations into actionable work. These practices are essential when teams need to balance feature velocity with service stability.

What Makes an SLO Effective

An effective SLO must be:

  1. Measurable over time
  2. Defined against a specific user experience or system behavior
  3. Connected to a quantifiable error budget that can drive decisions

SLOs do not exist in isolation. They must be grounded in real user impact, such as "95% of requests return within 200ms" rather than "system uptime is >99%". The SLO must include a clear window for measurement, typically 30 days.

Example configuration for an SLO:

slo:
  name: "API Response Time"
  description: "95th percentile of API responses under 200ms"
  objective: 0.95
  window: 30d
  metric:
    type: latency
    threshold: 200ms

The key here is that the SLO defines what constitutes a failure in terms of user experience, not system health alone.

Creating and Tracking Error Budgets

An error budget is the amount of "allowed" failure for a given SLO. It's calculated as 1 - objective. For an SLO with 95% objective, the error budget is 5%.

Error budgets are tracked by calculating how much of the allowed failure has been consumed during a time window.

To compute error budget consumption:

error_budget_consumed = (total_failures / total_requests) / (1 - objective)

If you have a 95% SLO and observe 6% failure rate, your error budget is fully consumed.

Error budgets should be tracked using a simple counter or a dedicated monitoring system that supports time-based calculations.

Setting Alerting Rules for SLOs

Alerting based on SLOs requires different logic than traditional metrics. You must alert not only when the SLO is failing but also when you are approaching the error budget limit.

For example, if you have a 95% SLO and 20% of your error budget is consumed (meaning 1.5% of requests are failing), you should trigger a warning-level alert.

Alerting rules must account for:

Example configuration for alerting:

alert:
  name: "SLO Budget Alert"
  condition:
    budget_consumed > 0.20
    window: 1h
  severity: warning
  message: "Error budget consumed at 20% for API Response Time"

Using SLO Data to Make Engineering Decisions

SLOs should drive decisions about when to pause feature development, invest in reliability improvements, or release changes. When the error budget is fully consumed, teams must stop adding new features until reliability improves.

Engineering teams can use SLOs to:

When an SLO violation occurs, you should be able to trace it back to specific code paths or system behaviors. This means tracking which service, endpoint, or component is responsible for the failure.

Practical Implementation with Standard Tools

Use curl to test SLO compliance by making repeated requests and measuring response times:

for i in {1..100}; do
  curl -w "%{time_total}\n" -o /dev/null -s http://example.com/api/endpoint
done | sort -n | awk 'NR==1{min=$1} END{print "Minimum:", min}'

Use psql to query error budget consumption from a monitoring database:

SELECT
  slo_name,
  window_start,
  window_end,
  (total_failures::float / total_requests) / (1 - objective) AS budget_consumed
FROM slos
WHERE window_start > NOW() - INTERVAL '30 days'
ORDER BY window_start DESC;

Common Pitfalls and Silent Failures

The most common mistake is using SLOs that are not tied to user impact. For example, "system uptime > 99%" is a poor SLO because it doesn't reflect how users experience the service.

Another failure mode is ignoring the time window in calculations. If you calculate an SLO over a single day instead of 30 days, the results can be misleading.

A third pitfall is failing to update SLOs when user expectations change. A service that used to serve 100 requests per second but now handles 1000 might need different SLOs even if the absolute failure rate stays the same.

SLO Violation Response

When an SLO violation occurs, you should:

  1. Identify the root cause
  2. Determine if it's a one-off or systemic issue
  3. Decide whether to pause new feature development
  4. Document the incident for future planning

You can use a simple script to trigger alerts when error budgets are consumed:

#!/bin/bash
BUDGET_THRESHOLD=0.20
CURRENT_BUDGET=$(psql -t -A -F',' -c "SELECT (total_failures::float / total_requests) / (1 - objective) FROM slos WHERE name = 'API Response Time';")

if (( $(echo "$CURRENT_BUDGET > $BUDGET_THRESHOLD" | bc -l) )); then
  echo "Alert: Error budget consumed at $CURRENT_BUDGET"
fi

Integrating SLOs into CI/CD

SLOs can be integrated into continuous integration and deployment pipelines by making release decisions conditional on error budget health.

If the error budget for a service is fully consumed, new deployments should not be allowed until the budget is replenished or the team has decided to proceed despite the risk.

Measuring SLO Impact Over Time

Track how SLOs change as your system scales. As you add more features or users, your SLO targets may need adjustment.

Use historical data to understand when and why SLOs were violated, which helps inform future engineering decisions.

Conclusion

SLOs are most effective when they are tied to user experience, clearly defined, and used to make concrete engineering decisions. They should not be abstract measurements but tools that guide how teams work. Error budgets provide a practical way to measure and manage reliability risk, ensuring that stability remains a priority even as features are added.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.