Sre Platform Tooling

Production-grade guide to sre platform tooling covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.

This reference describes the tools and mechanisms used to build, operate, and maintain platform infrastructure from an SRE perspective. It covers how to implement observability, configuration management, deployment automation, and service discovery using standard tooling patterns. These are the systems that allow teams to treat infrastructure as code and automate operational workflows without depending on vendor-specific interfaces or abstractions.

Observability Stack Setup

To configure a metrics collection system, start by creating a Prometheus configuration file with static targets:

scrape_configs:
  - job_name: 'service-metrics'
    static_configs:
      - targets: ['localhost:9090', 'backend:8080']

Prometheus scrapes metrics from configured endpoints at regular intervals. A common failure mode occurs when a target becomes unreachable but Prometheus continues to report the last known state. The scrape_timeout parameter controls how long Prometheus waits for a response before marking a scrape as failed.

For logs aggregation, configure a log collector like Fluentd with an output plugin for your storage system:

<match **>
  @type elasticsearch
  host localhost
  port 9200
</match>

A misconfigured log collector may silently drop entries when the backend is unavailable. The retry_limit parameter controls how many times it attempts to re-deliver dropped messages.

Infrastructure as Code

Manage platform configuration using GitOps workflows with declarative manifests. A Kubernetes deployment manifest might include:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: app
        image: nginx:latest
        ports:
        - containerPort: 80

Changes to this manifest must be committed to Git and applied via kubectl apply -f deployment.yaml. A common error occurs when kubectl apply is run without first checking for drift from the desired state.

Deployment Automation

Implement continuous deployment using a CI/CD pipeline. The simplest approach uses a bash script with kubectl set image:

#!/bin/bash
kubectl set image deployment/frontend frontend=nginx:1.21

This command updates the deployment's container image and triggers a rolling update. If the new image fails to start, the rollout will pause and require manual intervention.

A failure occurs when the pipeline does not wait for the rollout to complete before proceeding. Use kubectl rollout status to verify that all pods have been updated:

kubectl rollout status deployment/frontend

Service Discovery and Load Balancing

Configure service discovery using DNS records or Kubernetes service definitions. A service definition in Kubernetes:

apiVersion: v1
kind: Service
metadata:
  name: frontend-service
spec:
  selector:
    app: frontend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

When creating a service, ensure that the selector matches the labels on the pods. A mismatch causes traffic to be sent to no pods, resulting in silent failures.

Health Checks and Readiness Probes

Define readiness probes in Kubernetes deployments:

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

The initialDelaySeconds controls how long the pod waits before starting to check readiness. If this value is too short, pods may be marked ready before they are actually serving requests.

Configuration Management

Store application configuration in Kubernetes ConfigMaps:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  database_url: "postgres://localhost:5432/myapp"
  log_level: "info"

Mount this ConfigMap into a pod:

envFrom:
- configMapRef:
    name: app-config

Misconfiguration occurs when a ConfigMap key is misspelled in the application code, causing runtime errors.

Rollback and Versioning

Use Git tags to track deployment versions:

git tag v1.2.3
git push origin v1.2.3

When rolling back, reference the tag:

kubectl set image deployment/frontend frontend=nginx:v1.2.3

A common mistake is using a version that doesn't exist in the container registry, causing pods to enter a CrashLoopBackOff state.

Monitoring Alerting Rules

Create alerting rules in Prometheus:

groups:
- name: service-alerts
  rules:
  - alert: HighErrorRate
    expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.01
    for: 2m
    labels:
      severity: page

The for duration determines how long a condition must persist before triggering an alert. Setting this too low creates noise; setting it too high delays critical notifications.

Resource Management

Set resource limits and requests in Kubernetes:

resources:
  requests:
    memory: "64Mi"
    cpu: "250m"
  limits:
    memory: "128Mi"
    cpu: "500m"

If a pod exceeds its memory limit, it will be killed by the OOM killer. If CPU limits are too restrictive, performance may degrade.

Debugging and Diagnostics

Use kubectl describe to get detailed information about pods:

kubectl describe pod frontend-7b5b8c9d4f-xyz12

This command shows events, conditions, and container statuses. A common issue is missing or incomplete event logs that obscure root causes.

Failure Mode Detection

Implement circuit breaker patterns in your applications to prevent cascading failures. When a service becomes unavailable, the circuit breaker should transition to the open state and reject requests for a period:

circuit := &gobreaker.CircuitBreaker{
  Name: "service-circuit",
  ReadyToTrip: func(counts gobreaker.Counts) bool {
    return counts.ConsecutiveFailures > 3
  },
}

The circuit breaker will reset after a timeout period. A failure occurs when the timeout is too short, causing frequent resets and failing to protect the downstream service.

Scaling and Auto-Scaling

Configure horizontal pod autoscaling:

kubectl autoscale deployment frontend --cpu-percent=70 --min=2 --max=10

This creates a HorizontalPodAutoscaler that scales based on CPU usage. The --min and --max flags define the range of replicas.

A common issue is scaling too slowly, causing performance degradation during traffic spikes. Set appropriate targetCPUUtilizationPercentage values to ensure timely response.

Backup and Recovery

Schedule regular backups using cron jobs:

0 2 * * * /usr/local/bin/backup.sh

Ensure that backup scripts handle errors properly and log completion status. A failure in the backup process may silently succeed, leaving data unrecoverable.

Testing Infrastructure Changes

Use kubectl drain to safely remove nodes from service:

kubectl drain node-1 --ignore-daemonsets

This command cordons the node and evicts all pods. A misconfigured drain can cause unexpected downtime if pods are not properly configured for graceful shutdown.

Integration Points

Connect your platform tools through standard interfaces. For example, integrate a monitoring system with an alerting tool using webhooks:

webhook:
  url: "https://alerting.example.com/webhook"
  method: POST

Ensure that HTTP responses are handled correctly. A failing webhook endpoint can cause alerts to be lost.

Platform Integration Patterns

Use standard authentication mechanisms when connecting services. For example, authenticate with a database using connection strings:

psql "postgresql://user:password@host:port/database"

A misconfigured connection string may silently fail or use default credentials, leading to access issues.

Deployment Rollout Strategies

Implement blue-green deployments by maintaining two identical environments. Switch traffic between them using service updates:

kubectl patch service frontend -p '{"spec":{"selector":{"version":"green"}}}'

This command updates the service selector to route traffic to a different set of pods. A failure occurs when the new version is not properly tested before switching.

Resource Cleanup

Automate cleanup of unused resources using labels and selectors:

kubectl delete pods -l app=frontend,version=v1

Use --dry-run to preview deletions before applying them:

kubectl delete pods -l app=frontend --dry-run=client

A failure mode occurs when cleanup scripts do not handle edge cases properly, leaving resources in inconsistent states.

Version Compatibility

Maintain compatibility between tool versions by pinning dependencies in your CI/CD pipeline:

image: kubectl:v1.21.0

Version mismatches can cause subtle failures or unexpected behavior. Always test changes in staging before deploying to production.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.