Production-ready guide covering reliability review process for new services with implementation patterns, code examples, and anti-patterns for enterprise engineering teams.
This process defines how to evaluate and approve new service deployments for production reliability before they are made available to users. It applies when a service is ready for its first production deployment or when a significant change introduces new reliability risks that were not present in the existing service.
Before deploying a new service, examine the service definition and configuration files for critical reliability properties.
The service must define an explicit health check endpoint. For example, a typical HTTP-based service should expose a /healthz path that returns HTTP 200 when the service is ready to serve traffic.
curl -f http://service.example.com/healthzA failed health check causes the load balancer to remove the instance from rotation. The service must also define a readiness endpoint, typically /readyz, which returns HTTP 200 only when the service is fully initialized and can handle requests.
curl -f http://service.example.com/readyzThe health check should respond within 100ms under normal conditions. A timeout of more than 5 seconds causes the system to mark the instance as unhealthy.
Services must specify resource requirements explicitly in their deployment manifest. The CPU limit should be set to prevent a single service from consuming all available resources on a node.
resources:
limits:
cpu: "500m"
memory: "512Mi"
requests:
cpu: "100m"
memory: "128Mi"The ratio between limits and requests must be at least 2:1 for CPU and 4:1 for memory to ensure predictable scheduling behavior.
Services must declare all dependencies explicitly. For database connections, the service should define a maximum connection pool size:
-- Example PostgreSQL connection configuration
max_connections = 100The service must include a connection timeout of at least 5 seconds:
psql -d database_name -c "SELECT 1;" --connect-timeout=5Services using external APIs must define retry policies with exponential backoff and circuit breaker logic.
The readiness probe should be configured to check the service's ability to handle requests. The probe should use HTTP GET against the /readyz endpoint with a timeout of 3 seconds:
readinessProbe:
httpGet:
path: /readyz
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3A readiness probe failure causes the service to stop receiving traffic even if it is running.
The liveness probe must be separate from the readiness probe and should check that the service process is alive:
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 3The liveness probe should have a longer timeout than the readiness probe because it's checking process health, not request handling.
Services must register with the service discovery system before becoming available. The registration should include a TTL (time-to-live) value that is half the probe interval:
# Example registration call to service discovery
curl -X POST http://service-discovery.example.com/register \
-H "Content-Type: application/json" \
-d '{
"name": "my-service",
"port": 8080,
"ttl": 5
}'If the registration fails or the TTL expires, the service is removed from discovery.
Before production deployment, services must pass load tests with at least 100 concurrent users for 30 minutes. The test should include:
ab -n 1000 -c 100 http://service.example.com/api/testThe service must maintain sub-second response times under concurrent load. Failures in this step require a new review.
Services must be tested with failure injection to validate error handling:
# Example database connection failure simulation
curl -X POST http://service.example.com/failure/inject \
-d '{"type": "database_timeout"}'The service should log errors to the structured logging system and return appropriate HTTP status codes.
Services must implement graceful shutdown handling:
# Example shutdown signal handling
kill -TERM $PIDThe service must complete processing of in-flight requests before terminating. A timeout of 30 seconds is allowed for graceful shutdown, after which the process is killed with SIGKILL.
Services must expose metrics to the monitoring system at a /metrics endpoint. The following metrics are required:
curl http://service.example.com/metricsThe metrics endpoint should respond within 100ms.
Services must define alerting thresholds in their configuration. Critical alerts should be defined for:
The alerting system should support a 5-minute evaluation window.
Services must emit structured logs to the logging system. The log format should include:
{
"timestamp": "2023-10-01T10:00:00Z",
"service": "my-service",
"request_id": "abc123",
"level": "error",
"message": "Database connection failed"
}Before final approval, verify that all configuration entries are present and correct:
The deployment manifest must include:
All test results must be documented in the service's deployment checklist. This includes:
The most common failure is a readiness probe that returns HTTP 200 before the service is actually ready to handle requests. This causes traffic to arrive at an unprepared service, resulting in request failures.
When CPU limits are set too low, services may be throttled even under normal load, causing increased latency and reduced throughput.
Services that do not declare all dependencies fail during production deployment when unexpected external services become unavailable.
If service registration with discovery fails, the service becomes available to users but is not discoverable by other services.
Services without proper metrics or logging fail to provide visibility into their behavior in production, making incident response difficult.
Incorrect timeout configurations cause services to appear unhealthy even when they are functioning normally.
The reliability review must be completed by the service owner and reviewed by a senior SRE. The review includes:
Once approved, the service may proceed to production deployment. Rejection requires rework and re-submission of the review.
After deployment, the service must be monitored for:
If any critical issues appear during this period, the service may be rolled back or require additional reliability work.
This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy