Site Reliability Engineering (SRE) Practices

Implement SRE practices. Covers error budgets, incident management, blameless postmortems, capacity planning, toil reduction, and building reliability into engineering culture.

Site Reliability Engineering (SRE) Practices is a set of engineering disciplines applied to large-scale, production systems to ensure they are reliable, scalable, and maintainable. It matters when teams need to reduce toil, define service-level objectives, automate operations, and respond effectively to incidents across complex, distributed environments. This page details the core practices of SRE, focusing on what engineers actually do—how they design, monitor, deploy, and evolve systems—using real tools and well-established patterns.

Defining and Tracking Reliability

Reliability in SRE begins with clear, measurable objectives. A service is not reliable just because it’s up—it is reliable when it meets defined expectations over time. The foundation is the Service Level Objective (SLO), which quantifies reliability in terms of availability, latency, error rate, or throughput.

An SLO is defined as a target value for a key metric over a defined window. For example, a web API must return 99.9% of requests within 200ms, measured over a rolling 7-day window. This is expressed in a standard SLO definition:

slo:
  service: "user-profile-api"
  goal: 0.999
  window: "7d"
  metric: "latency"
  target: "p95"
  threshold: 200

This configuration is consumed by a monitoring system and used to calculate real SLO attainment daily. The Error Budget is derived from the SLO: if the SLO is 99.9%, the error budget is 0.1%, meaning 1000 failed requests per million over a 7-day period. Every error consumes a portion of the budget, and when the budget is exhausted, teams may allow new features or deployments.

SLOs are tracked with SLO dashboards that show:

Failure occurs silently when error budget is exhausted but no alert is triggered. The common mistake is to define SLOs without automating their calculation. The SLO engine must:

A typical failure: an SLO calculation job runs hourly but misses a 10-minute data window due to a dropped Kafka message. The SLO appears healthy, but a 1-hour outage went unnoticed because data was not ingested.

Automating Incident Response

An incident is not just a downtime event—it is a process. SREs define incident response workflows that begin when a service fails to meet its SLO or triggers a threshold alert.

An incident begins when a monitoring system detects a metric crossing a threshold. For example, a Prometheus alert rule:

groups:
  - name: "api-latency"
    rules:
      - alert: "HighLatency"
        expr: "rate(http_request_duration_seconds_sum{job='user-profile-api'}[5m]) / rate(http_request_duration_seconds_count[5m]) > 200"
        for: "10m"
        labels:
          severity: "warning"
        annotations:
          summary: "API latency exceeds 200ms for 10 minutes"
          description: "The 95th percentile latency for the user-profile-api has been above 200ms for 10 consecutive minutes."

The alert triggers a ticket in a system like Jira or GitHub Issues, with a standard format including:

The incident is then owned by a team or individual. The first responder runs a runbook—a series of documented steps to diagnose and resolve common issues.

A runbook entry for "HighLatency" might include:


# Step 1: Check service health
curl -s http://user-profile-api:8080/health | jq .

# Step 2: Check database latency
psql -c "SELECT avg(query_time) FROM traces WHERE service = 'user-profile-api' AND duration > 500"

# Step 3: Check recent deployments
git log --since="1h" --oneline --grep="deploy"

# Step 4: Roll back to previous deployment
kubectl rollout undo deployment/user-profile-api --to-revision=12

# Step 5: Notify team
curl -X POST https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX \
     -H 'Content-Type: application/json' \
     -d '{
           "text": "Incident INC-123: HighLatency resolved. Rolled back to revision 12.",
           "channel": "#sre-alerts"
         }'

A common failure: the runbook assumes kubectl is available in the incident response environment, but the team forgot to install it on the incident response VM. The second step fails silently. Another failure: the Slack notification uses a webhook URL that expires every 30 days, but the team missed the renewal.

Reducing Toil with Automation

Toil is any repetitive, manual, operational work. SREs reduce toil by automating tasks into runbooks, scripts, and pipelines.

A typical toil item is log rotation. The system generates 50GB of logs per day, and logs older than 30 days are moved to cold storage.

The automation is defined in a cron job:

# /etc/cron.daily/log-rotation
#!/bin/bash
set -e

LOG_DIR="/var/log/user-profile-api"
BACKUP_DIR="/mnt/cold-storage/logs"
DATE=$(date +%Y-%m-%d)

# Compress and move logs older than 30 days
find "$LOG_DIR" -name "*.log" -mtime +30 -exec gzip {} \; -exec mv {} "$BACKUP_DIR" \;

# Clean up old backups
find "$BACKUP_DIR" -name "*.log.gz" -mtime +365 -delete

# Notify completion
echo "Log rotation completed on $(date)" | mail -s "Log Rotation" [email protected]

This script runs daily, but silently fails when the backup directory is full. The mv command fails, but no error is logged. The solution is to add set -e and exec with error handling:

find "$LOG_DIR" -name "*.log" -mtime +30 \
  -exec sh -c 'for file; do
    gzip "$file" && mv "$file.gz" "$BACKUP_DIR"
  done' _ {} +

This ensures all files are processed or rolled back on error.

Managing Change with Deployment Practices

Deployments are the heartbeat of SRE. A change is not just a new version—it is a controlled, measurable process with rollback and verification.

A canary deployment is used to roll out new features with reduced risk. The deployment process is orchestrated via a CI/CD pipeline, which includes:

  1. Build the container image
  2. Push to registry
  3. Deploy to staging
  4. Run integration tests
  5. Deploy to production with canary rollout
  6. Verify SLOs for canary
  7. Promote to full rollout if SLOs are met

The canary deployment is defined in a Kubernetes manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-profile-api-canary
  labels:
    app: user-profile-api
    track: canary
spec:
  replicas: 4
  selector:
    matchLabels:
      app: user-profile-api
      track: canary
  template:
    metadata:
      labels:
        app: user-profile-api
        track: canary
    spec:
      containers:
        - name: api
          image: registry.company.com/user-profile-api:v2.1.0-canary
          ports:
            - containerPort: 8080
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
            limits:
              memory: "512Mi"
              cpu: "500m"
          env:
            - name: ENV
              value: "staging"
---
apiVersion: v1
kind: Service
metadata:
  name: user-profile-api-canary
spec:
  selector:
    app: user-profile-api
    track: canary
  ports:
    - port: 80
      targetPort: 8080
  type: ClusterIP

The canary is exposed via a traffic-splitting service—a load balancer or ingress controller that routes 10% of traffic to the canary version. After 2 hours, the system checks whether the canary SLOs are met:

If all SLOs are met, the deployment is promoted with:

kubectl patch deployment user-profile-api-canary -p='{"spec":{"replicas":16}}'
kubectl rollout status deployment user-profile-api-canary

The promotion can be automated via a post-deployment hook in the CI/CD tool:

- name: "Promote canary to production"
  run: |
    # Check SLOs via API
    SLO_RESPONSE=$(curl -s http://slo-engine:8080/api/slo/user-profile-api-canary)
    if echo "$SLO_RESPONSE" | jq -e '.slo > 0.995 and .error_rate < 0.001'; then
      echo "SLOs met. Promoting to production."
      kubectl patch deployment user-profile-api -p='{"spec":{"replicas":16}}'
      kubectl rollout status deployment user-profile-api
    else
      echo "SLOs failed. Rolling back."
      kubectl rollout undo deployment user-profile-api-canary
    fi

A subtle failure: the SLO engine returns JSON with slo as a float, but the CI job expects slo_value. The jq -e command fails silently because slo is not present. The fix is to use jq -e '.slo_value // .slo'.

Managing Configuration and State

Configuration is a critical part of SRE. It governs how systems behave, and changes to configuration must be versioned, tested, and deployed.

A common configuration pattern uses environment-specific configuration files stored in version control:

// config/staging.json
{
  "database": {
    "host": "db-staging.company.com",
    "port": 5432,
    "pool_size": 10
  },
  "logging": {
    "level": "info",
    "format": "json"
  },
  "slo": {
    "window": "7d",
    "goal": 0.999
  }
}

These files are applied to services via configuration management—tools like Ansible, SaltStack, or custom scripts.

A typical configuration deployment script:

#!/bin/bash
set -euo pipefail

SERVICE_NAME=$1
ENVIRONMENT=$2
CONFIG_DIR="./config"

# Validate environment
if [[ ! -f "${CONFIG_DIR}/${ENVIRONMENT}.json" ]]; then
  echo "Configuration file not found: ${CONFIG_DIR}/${ENVIRONMENT}.json"
  exit 1
fi

# Apply configuration
mkdir -p "/etc/${SERVICE_NAME}"
cp "${CONFIG_DIR}/${ENVIRONMENT}.json" "/etc/${SERVICE_NAME}/config.json"
chown -R app:app "/etc/${SERVICE_NAME}"
chmod 644 "/etc/${SERVICE_NAME}/config.json"

# Restart service
systemctl reload "${SERVICE_NAME}.service"

# Log success
echo "Configuration applied for ${SERVICE_NAME} in ${ENVIRONMENT}"

The script fails silently when the systemctl reload command is not available. The solution: use systemctl only if it exists, otherwise fall back to service.

Another failure: the config.json file is not idempotent. A second run of the script overwrites the file but does not preserve user changes. The fix is to merge new configuration with existing:

# Merge new config with existing
jq --argjson new_config "$(cat config.json)" '. * $new_config' /etc/${SERVICE_NAME}/config.json > /tmp/config.json
mv /tmp/config.json /etc/${SERVICE_NAME}/config.json

This ensures that environment-specific overrides are preserved across deployments.

Evolving Systems with Feedback Loops

SRE practices are not static. They rely on feedback loops that close the gap between operation and development.

A key loop is the SLO-to-Deployment loop:

This loop is powered by event-driven pipelines that react to SLO data.

For example, when the error budget for user-profile-api is below 0.05% for three consecutive days, the system triggers a feature deployment:

# Trigger feature deployment when error budget < 0.05%
curl -X POST http://pipeline-engine:8080/api/pipeline/trigger \
     -H 'Content-Type: application/json' \
     -d '{
           "pipeline": "user-profile-api-feature-deploy",
           "reason": "Error budget below 0.05%",
           "parameters": {
             "branch": "main",
             "release_version": "v2.2.0"
           }
         }'

The pipeline then builds, tests, and deploys the new version. After deployment, it updates the SLOs and notifies stakeholders.

The loop is broken when SLO data is not updated after deployment. The error budget is calculated from old data, and the trigger condition is based on stale metrics. The fix is to update the SLO engine after every deployment, ensuring that the feedback loop is closed.

SRE practices are the glue between software and operations. They are not a set of tools but a way of working—measurable, automated, and continuous. The best SRE teams do not just run systems; they evolve them through feedback, reliability, and deep understanding of what it means to deliver software at scale.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.