SLO Engineering

Define, measure, and manage Service Level Objectives that align engineering priorities with user expectations. Covers SLI selection, error budget policy, SLO-based alerting, and the organizational process that makes SLOs actionable.

SLO Engineering is the practice of designing, configuring, and maintaining Service Level Objectives that are operational, measurable, and actionable. It matters when teams need to align on what “good service” means, when a service exceeds expectations, and when they must detect and respond to degradation before users feel it. It is the foundation of observability-driven operations, where metrics, logs, and traces are not just collected but are tied to concrete, measurable commitments.

Defining SLOs

An SLO begins with a clear, quantifiable goal: a target for a service’s performance or reliability. The definition must specify the metric, the time window, and the target value.

The most common SLO metric is *availability*, defined as the ratio of successful requests to total requests over a time window. For a REST API, this is:

SLO = (successful requests in last 24h) / (total requests in last 24h)

To express this as a service-level objective, define it in a configuration file. Use a standard format—YAML, JSON, or plain text—without assuming a particular platform.


- name: api-availability
  metric: requests
  filter: service=api
  window: 24h
  goal: 0.999
  error_budget: 0.001
  threshold: 99.9

The goal field is the target success rate, expressed as a decimal (e.g., 0.999 for 99.9% availability). The error_budget is the maximum allowable failure rate, derived from the goal. The threshold is the human-readable form used in dashboards and alerts.

When defining an SLO, engineers must decide on the *time window*—the duration over which measurements are aggregated. Common windows are 1h, 6h, 12h, 24h, 7d, and 30d. A 24h window is typical for daily service health reviews. A 7d window captures weekly trends and is useful for quarterly SLO reviews.

Configuring SLO Targets

SLOs are not static. They evolve with service growth, user expectations, and operational maturity. To support this, SLOs can include *tiered goals* and *time-based targets*.

For example, a service might have a primary goal of 99.9% availability over 24h, but a secondary goal of 99.99% availability over 7d. This allows for a more nuanced error budget.

- name: api-availability
  metric: requests
  filter: service=api
  window: 24h
  goals:
    - target: 0.999
      window: 24h
    - target: 0.9999
      window: 7d
  error_budget:
    - window: 24h
      initial: 0.001
    - window: 7d
      initial: 0.0001

The goals array allows multiple performance tiers. The error_budget object defines the initial error budget for each window. These budgets are consumed over time and can be visualized as a “burning” graph.

When a service has distinct user segments, SLOs can be defined per *user tier*.

- name: premium-user-latency
  metric: latency
  filter: service=api, user_type=premium
  window: 1h
  goal: 0.95
  threshold: 95
  unit: milliseconds
  target: 200

This SLO defines that 95% of premium user requests must complete within 200ms over a 1-hour window.

Calculating SLO Measurements

SLOs depend on accurate measurement of their underlying metric. The measurement process involves collecting raw data, aggregating it, and computing the ratio or distribution.

For availability, the raw data is a stream of request events, each with a timestamp, status, service, and duration. Tools like Prometheus, Grafana, or a custom logging pipeline must emit this data in a structured format.

To compute the SLO value at a given time, use a query or script that:

  1. Filters events for the relevant service and time window.
  2. Counts the number of successful (status >= 200 and < 400) and total requests.
  3. Computes the ratio.

In SQL, this is:

SELECT
  COUNT(*) FILTER (WHERE status >= 200 AND status < 400) AS successful,
  COUNT(*) AS total,
  (COUNT(*) FILTER (WHERE status >= 200 AND status < 400) * 1.0 / COUNT(*)) AS sloscore
FROM requests
WHERE service = 'api'
  AND timestamp >= NOW() - INTERVAL '24 hours'
GROUP BY 1;

For latency SLOs, the computation is more involved. The metric is a distribution of request durations. The SLO goal is the fraction of requests below a threshold.

SELECT
  COUNT(*) FILTER (WHERE duration <= 200) AS within_threshold,
  COUNT(*) AS total,
  (COUNT(*) FILTER (WHERE duration <= 200) * 1.0 / COUNT(*)) AS sloscore
FROM requests
WHERE service = 'api'
  AND user_type = 'premium'
  AND timestamp >= NOW() - INTERVAL '1 hour'
GROUP BY 1;

The duration column is in milliseconds. The threshold is 200ms.

Evaluating SLO Conformance

Once SLOs are defined and measured, the next step is to evaluate whether the service is meeting its SLOs over a period.

An SLO is *conforming* if its measured value meets or exceeds the goal. The evaluation is done over a fixed period—typically one day, one week, or one month.

A daily evaluation uses the window and goal fields to determine conformance.

# Evaluate SLO for today
curl -X POST https://slo-engine/api/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "slo": "api-availability",
    "start": "2024-04-05T00:00:00Z",
    "end": "2024-04-05T23:59:59Z"
  }'

The response includes the measured SLO value, the goal, and a conformance flag:

{
  "slo": "api-availability",
  "window": "24h",
  "start": "2024-04-05T00:00:00Z",
  "end": "2024-04-05T23:59:59Z",
  "measured": 0.9987,
  "goal": 0.999,
  "conforms": false,
  "error_budget_used": 0.0003
}

The conforms field is true if measured >= goal. The error_budget_used is the fraction of the initial error budget consumed. A service using 30% of its error budget over 24h is considered to be on track.

Managing Error Budgets

The error budget is the central mechanism for SLO-driven operations. It is the allowance for service failure, expressed as a time or probability.

Every SLO has an initial error budget. For a 99.9% availability SLO, the error budget is 0.001. Each day, the service consumes a portion of this budget based on its actual performance.

The error budget is consumed over time. A service that is 99.8% available on a day consumes 0.002 of error budget—twice its daily allowance.

To track error budget consumption, use a time-series database or a simple table.

CREATE TABLE error_budget (
  slo_name TEXT PRIMARY KEY,
  window TEXT NOT NULL,
  initial_budget DOUBLE PRECISION NOT NULL,
  consumed_budget DOUBLE PRECISION DEFAULT 0,
  last_updated TIMESTAMP WITH TIME ZONE
);

-- Update error budget after each SLO evaluation
UPDATE error_budget
SET consumed_budget = consumed_budget + 0.0003,
    last_updated = NOW()
WHERE slo_name = 'api-availability'
  AND window = '24h';

When the error budget is fully consumed, the service is in *error budget exhaustion*. This triggers operational actions: increased monitoring, alerting, or even a shift to a higher maintenance mode.

Integrating SLOs into Alerting

SLOs are not just metrics—they are alerts. The most common alerting pattern is *burn rate alerting*.

A burn rate alert triggers when the error budget is being consumed faster than expected.

# alerting.yaml
- name: api-availability-burn-rate
  type: burn_rate
  slos:
    - api-availability
  window: 7d
  threshold: 1.5
  message: |
    Error budget burn rate exceeds 1.5x expected.
    Current: {{current_burn_rate}}
    Expected: {{expected_burn_rate}}
    Budget used: {{budget_used}}
  severity: warning

The burn_rate alert compares the actual error budget consumption to the expected rate. The expected rate is derived from the SLO goal and the average performance.

To compute burn rate:

  1. Compute the total error budget used over the window.
  2. Divide by the number of days in the window.
  3. Divide by the expected daily consumption.
WITH slos AS (
  SELECT
    slo_name,
    window,
    SUM(measured) AS total_measured,
    COUNT(*) AS days
  FROM slos_daily
  WHERE slo_name = 'api-availability'
    AND start >= NOW() - INTERVAL '7 days'
  GROUP BY 1, 2
),
expected AS (
  SELECT
    slo_name,
    window,
    initial_budget / 7 AS expected_daily
  FROM slos
  WHERE window = '24h'
)
SELECT
  s.slo_name,
  s.window,
  (s.total_measured / s.days) / e.expected_daily AS burn_rate
FROM slos s
JOIN expected e ON s.slo_name = e.slo_name;

When the burn rate exceeds the threshold (e.g., 1.5), the alert triggers. The alert includes details about the current and expected burn rate, and the total error budget consumed.

SLOs in Change Management

SLOs are central to change management. Every deployment or configuration change should be evaluated against SLOs.

The process is: deploy a change, measure SLOs over a period, and decide whether the change succeeded.

A change is *successful* if the SLOs meet or exceed the goal during the change window.

# Deploy a new version
curl -X POST https://deploy/api/v1/deploy \
  -H "Content-Type: application/json" \
  -d '{
    "service": "api",
    "version": "v2.3.1",
    "change": "enable caching for user profile requests"
  }'

# Wait 30 minutes
sleep 30m

# Evaluate SLOs for the change period
curl -X POST https://slo-engine/api/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "slo": "api-availability",
    "start": "2024-04-05T12:00:00Z",
    "end": "2024-04-05T12:30:00Z"
  }'

The evaluation returns the SLO value. If the service is 99.92% available, it exceeds the goal of 99.9%, and the change is considered successful.

Failure modes include:

When a change fails, the team can roll it back or investigate further.

Common Pitfalls and Sharp Edges

The most common failure is a mismatch between the SLO definition and the actual measurement. The SLO might define “success” as status >= 200 AND < 400, but the actual data includes status = 400 as failure, even though 400 is a valid success code.

Another edge case: the time window overlaps with the next evaluation. If SLOs are evaluated daily at midnight, and a change is deployed at 11:55 PM, the first evaluation might include both the old and new versions.

SLOs are often misconfigured with incorrect window or goal. A 99.9% SLO over a 12-hour window is not equivalent to one over 24 hours. The error budget must be adjusted accordingly.

The threshold field is often misunderstood. It is not the same as the goal. The goal is the target value (e.g., 0.999), while the threshold is the human-readable form (e.g., 99.9). The system must convert between them consistently.

Finally, error budget consumption is not always additive. When multiple SLOs are defined, their error budgets can be independent or shared. A shared error budget requires a mechanism to track and allocate budget across SLOs.

In practice, teams often forget to update the initial value of the error budget after a change. A service that was 99.9% available for a month has a remaining error budget, but the new SLO configuration starts from zero.

The silent failure: a service is evaluated over a period, but the evaluation window does not align with the data collection window. If data is collected every 5 minutes but evaluations are done every 24 hours, the first evaluation might use only the last 23 hours of data.

Summary

SLO Engineering is the practice of grounding service reliability in measurable, actionable commitments. It begins with defining SLOs that are clear, consistent, and aligned with operational goals. It continues through accurate measurement, conformance evaluation, error budget management, and integration into alerting and change management. The key is not just having SLOs, but using them to drive decisions, detect failure early, and improve service quality over time.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.