Production-grade guide to site reliability anti patterns covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.
This page documents common failure modes in SRE practices that create brittleness, confusion, and operational debt. These patterns appear when teams build systems without considering how they will behave under stress, when alerts are misconfigured, or when operational workflows are not designed for scale and resilience.
A common anti-pattern is creating alert rules with thresholds that trigger frequently for minor variations. This creates alert fatigue and causes teams to ignore real issues.
Example alert configuration using Prometheus:
- alert: HighCPUUsage
expr: rate(container_cpu_usage_seconds_total{job="kubelet"}[5m]) > 0.8
for: 2m
labels:
severity: warningThis rule triggers on any container using more than 80% CPU for two minutes, even when the system is operating normally. The alert should instead use a sliding window or percentage-based alerting to distinguish between temporary spikes and sustained problems.
Alerts without sufficient context fail when operators must investigate manually. Consider an alert that only shows a metric name:
Alert: DatabaseConnectionPoolExhausted
Severity: criticalThis is insufficient. The proper alert should include database instance, connection count, and the time of last successful connection.
When alerting systems trigger other alerts without clear ownership boundaries, teams struggle to assign responsibility. For example, a monitoring system might send an alert to a team that owns infrastructure, but the root cause is in application code.
Teams that deploy manually without rollback capabilities create single points of failure. Example deployment workflow:
# This fails if the new version breaks unexpectedly
kubectl set image deployment/myapp myapp=registry.example.com/myapp:v2.1.0This command does not provide a built-in rollback mechanism. The operator must remember to save the previous image tag or manually perform a rollback.
Deployment tools that proceed despite failing readiness checks. Example Kubernetes deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
replicas: 3
template:
spec:
containers:
- name: app
image: registry.example.com/myapp:v2.1.0
readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 10If the readiness probe fails, the deployment continues. The correct behavior is to pause or rollback the deployment when readiness checks fail.
Teams that assume all deployments can be done without downtime, even for critical systems. For example, assuming a database schema change can be applied with zero downtime:
ALTER TABLE users ADD COLUMN email_verified BOOLEAN DEFAULT FALSE;This statement may lock the table for extended periods depending on table size and storage engine, causing service degradation or failure.
Teams that use different naming conventions for similar metrics across services create confusion. Example metric names:
http_requests_total
api_requests_count
request_count
requests_processedAll represent request volume but with no standardization. The correct approach is to define a consistent metric naming schema.
Counting errors in the wrong scope, such as counting HTTP 500s at the application level instead of the service level. Example incorrect approach:
@app.route('/api/users')
def get_users():
try:
users = db.query("SELECT * FROM users")
return jsonify(users)
except Exception as e:
# This counts individual failures but not service failures
error_counter.inc()
return jsonify({"error": "Database failure"}), 500The correct approach is to count errors at the service boundary, not at the individual request level.
Teams that focus only on average latency without considering percentiles create false confidence. Example monitoring script:
# This ignores tail latencies
curl -w "@curl-format.txt" -o /dev/null -s http://api.example.com/usersWhere curl-format.txt contains:
time_namelookup: %{time_namelookup}\n
time_connect: %{time_connect}\n
time_total: %{time_total}\nThis only shows the average but not how often requests take longer than expected.
Capacity planning that ignores resource contention between services. Example system where two applications share a database pool:
# Both applications use the same database connection pool
# But neither accounts for resource sharing
app1:
database_pool_size: 20
app2:
database_pool_size: 20If both applications are running, they may exhaust available connections even though each application is within its own limits.
Teams that assume capacity scales linearly with resources. For example, expecting that doubling CPU cores doubles throughput:
# This assumes linear scaling of database performance
kubectl scale deployment/database --replicas=2Database performance may not scale linearly due to locking, caching behavior, or I/O limitations.
Capacity planning that only considers average usage without peak patterns. Example:
# Only monitoring average CPU usage
kubectl top podsThis misses periods of high utilization that cause service degradation or failure.
Teams that hardcode values into production code instead of using configuration management. Example:
# This is hardcoded and cannot be changed without redeployment
def get_timeout_seconds():
return 30 # hardcoded valueThe correct approach is to load this from environment variables or configuration files.
Teams that maintain different configurations for development, staging, and production environments. Example:
# Different timeouts in different environments
development:
timeout: 5s
staging:
timeout: 10s
production:
timeout: 30sThis creates configuration drift where code behaves differently across environments.
Teams that deploy configurations without validating them. Example deployment process:
# No validation step
kubectl apply -f config.yaml
kubectl rollout status deployment/myappThis assumes the configuration is valid without checking for syntax errors or invalid values.
Teams that do not define clear escalation paths when incidents occur. Example:
Incident: API response times are slow
Action: Monitor metrics and wait for improvementThis lacks specific thresholds for when to escalate, who to contact, or what actions to take.
Teams that use chat-based incident management without maintaining logs. Example:
# Incident is handled via Slack but no structured record
!deploy production
!rollback v1.2.3This makes it difficult to analyze what happened or why certain decisions were made.
Teams that resolve incidents quickly but fail to perform root cause analysis. Example incident resolution:
Issue: Database connection failures
Resolution: Restarted database serviceThis does not investigate whether the problem was temporary or systemic, leaving teams vulnerable to recurrence.
Teams that create infrastructure with single points of failure. Example:
# This configuration creates a single point of failure
api_gateway:
replicas: 1
load_balancer: externalIf the single API gateway instance fails, the entire service becomes unavailable.
Teams that create deep dependency chains where failure in one component cascades through many others. Example:
# Complex chain of dependencies
service-a -> service-b -> service-c -> databaseIf service-c fails, it affects both service-a and service-b, even though service-a might not need service-c.
Teams that implement services without circuit breaker patterns. Example:
# This can cause cascading failures
def call_external_service():
response = requests.get("http://external-service/api/data")
return response.json()This function can fail completely if the external service is slow or unavailable, causing the calling service to also fail.
Teams that deploy without integration testing, relying only on unit tests. Example:
# Only unit tests, no integration tests
def test_api_endpoint():
# Tests individual function behavior
assert process_user_data(input_data) == expected_outputThis does not validate the full integration flow from API to database.
Teams that deploy without load testing, assuming systems will handle production volumes. Example:
# Deployment without load test validation
kubectl apply -f production.yaml
kubectl rollout status deployment/myappThis assumes the system can handle production load without prior validation.
Teams that implement data validation inconsistently across services. Example:
# Different validation rules in different services
def validate_email(email):
# Service A validation
return "@" in email
def validate_email(email):
# Service B validation
return re.match(r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$', email)This creates data quality issues when services have different expectations.
Teams that perform data migrations without considering impact to running services. Example:
# This migration does not account for service availability
psql -c "ALTER TABLE users ADD COLUMN last_login TIMESTAMP"This command may lock the table and prevent read/write operations during the migration.
Teams that implement background jobs without proper error handling. Example:
# This job silently fails on database errors
def process_queue():
while True:
try:
item = queue.get()
process_item(item)
except Exception:
# No logging, no retry, no notification
passThis creates situations where jobs silently fail without operator awareness.
Teams that return inconsistent error responses from APIs. Example:
# Different error formats in different endpoints
# API endpoint 1
{"error": "Database connection failed"}
# API endpoint 2
{"message": "Connection timeout"}This makes client-side error handling more complex and increases maintenance overhead.
Teams that deploy without proper resource limits, leading to resource starvation. Example:
# No resource constraints defined
apiVersion: v1
kind: Pod
metadata:
name: myapp
spec:
containers:
- name: app
image: registry.example.com/myapp:v2.1.0This creates potential for one pod to consume all available resources on a node.
Teams that do not configure proper cleanup policies for temporary resources. Example:
# This creates temporary files without cleanup
kubectl logs mypod > /tmp/logs.txtThis does not clean up temporary files, potentially filling disk space over time.
Teams that grant broad permissions by default. Example Kubernetes RBAC:
# This gives full cluster access to a namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: admin-binding
namespace: mynamespace
subjects:
- kind: ServiceAccount
name: default
namespace: mynamespace
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.ioThis grants full cluster access to a service account that only needs basic namespace permissions.
Teams that implement access control inconsistently. Example:
# Service A requires authentication for all endpoints
# Service B allows anonymous access to some endpoints
# Service C uses different token formatsThis creates confusion for clients and makes security harder to maintain.
Teams that deploy without standardized checklists, leading to missing steps. Example:
# No deployment checklist
kubectl apply -f config.yaml
kubectl set image deployment/myapp myapp=registry.example.com/myapp:v2.1.0
kubectl rollout status deployment/myappThis process omits important steps like pre-deployment health checks or post-deployment validation.
Teams that deploy without documented rollback procedures. Example:
# Deployment without rollback plan
kubectl set image deployment/myapp myapp=registry.example.com/myapp:v2.1.0This does not include how to roll back if the new version causes problems.
Teams that do not use standardized incident response templates, leading to inconsistent documentation and analysis. Example:
# Different formats for different incidents
# Incident 1:
# - Problem description
# - Resolution steps
# - Root cause analysis
# Incident 2:
# - What happened
# - How it was fixed
# - Lessons learnedThis makes it difficult to perform cross-incident analysis and improve processes.
This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy