Backend Health Check Patterns

Backend systems depend on health checks to detect failure before users notice. A health check is a lightweight, synchronous probe that validates a service’s readiness and liveness, typically via a simple HTTP endpoint. It runs continuously in orchestration platforms (Kubernetes, Nomad), load balancers, and service meshes, and it determines when a service is available to serve traffic and when it should be taken offline for maintenance or upgrades. This page details the patterns, practices, and pitfalls of implementing effective backend health checks across services, from monoliths to microservices.

Core Health Check Types

Readiness Probe

A readiness probe determines when a service is ready to accept traffic. It runs during startup and signals that the service has loaded dependencies, initialized caches, connected to databases, and completed any warm-up work.

Key pattern: The service exposes an /health/ready endpoint that returns 200 OK when ready, and 503 Service Unavailable when not. The probe checks this endpoint at regular intervals and only adds the service to the load balancer once it succeeds.

GET /health/ready HTTP/1.1
Host: api.example.com
Accept: application/json

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 21

{"status": "ready"}

The probe expects the following:

• status: "ready" (exact string)
• HTTP status 200
• No body required, but a structured body is common
• Response time under 200ms
• No 5xx or 4xx errors

When a readiness probe fails, the service is removed from the load balancer and stops receiving requests until it passes again.

Liveness Probe

A liveness probe determines whether a service is running and responsive. It is more aggressive than readiness and runs continuously during the service’s lifecycle.

Key pattern: The endpoint /health/live returns 200 OK if the service is alive, 500 Internal Server Error otherwise. The probe checks this endpoint periodically.

GET /health/live HTTP/1.1
Host: api.example.com
Accept: application/json

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 22

{"status": "alive", "timestamp": 1712345678}

The probe expects:

• status: "alive" (exact string)
• HTTP status 200
• timestamp field in ISO 8601 or Unix timestamp format
• Response time under 100ms
• No 5xx or 4xx errors

If a liveness probe fails, the orchestration system restarts the service process (e.g., kill -9 and exec).

Health Check with Dependency Validation

A service may depend on multiple external systems. A health check that only validates the service itself is insufficient.

Pattern: The /health endpoint returns a structured result indicating the status of all dependencies.

GET /health HTTP/1.1
Host: api.example.com
Accept: application/json

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 324

{
  "status": "ready",
  "details": {
    "database": {
      "status": "healthy",
      "latency_ms": 18,
      "error": null
    },
    "redis": {
      "status": "healthy",
      "latency_ms": 12,
      "error": null
    },
    "message_queue": {
      "status": "unhealthy",
      "latency_ms": 210,
      "error": "Connection timeout to broker.example.com:5672"
    }
  }
}

Key rules:

• Top-level status is one of "ready", "unhealthy", "degraded", "starting", "maintenance".
• details is a map of service names to status objects.
• Each status object includes:
• status: string ("healthy", "unhealthy", "warning", "pending")
• latency_ms: integer
• error: string (nullable), describing the failure
• The service returns 200 if overall status is "healthy" or "ready", 503 if "unhealthy" or "degraded", 500 if critical failure.

This pattern enables fine-grained alerting and gradual degradation strategies.

Probe Configuration and Timing

Probe Configuration in Kubernetes

Kubernetes uses readinessProbe and livenessProbe fields in Pod specs.

apiVersion: v1
kind: Pod
metadata:
  name: api-server
spec:
  containers:
    - name: api
      image: example/api:v1.4
      ports:
        - containerPort: 8080
      readinessProbe:
        httpGet:
          path: /health/ready
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 10
        periodSeconds: 5
        timeoutSeconds: 3
        failureThreshold: 3
      livenessProbe:
        httpGet:
          path: /health/live
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 30
        periodSeconds: 10
        timeoutSeconds: 5
        failureThreshold: 3

Critical parameters:

• initialDelaySeconds: time after pod start before first probe
• periodSeconds: frequency of checks
• timeoutSeconds: how long to wait for a response
• failureThreshold: number of consecutive failures before marking the service as ready/unhealthy
• successThreshold: number of successful probes to mark the service as ready (default: 1)

Failure Threshold and Tolerance

A service may become transiently unavailable due to GC, network jitter, or slow startup. The failure threshold determines how long a service must be down before it is marked as failed.

Example: A service with periodSeconds: 5, failureThreshold: 3:

• Probe runs every 5 seconds.
• First failure: probe returns 503 or times out.
• Second failure: total 10 seconds of failure.
• Third failure: 15 seconds of failure → service marked unhealthy.

During this window, the service may still receive traffic (if readiness is good), but liveness failures trigger restarts.

Timeout and Concurrency

A health check must respond within timeoutSeconds. If the service takes longer, the probe fails and marks the service as unavailable.

Common causes of timeout:

• Long GC pauses
• Database connection pool exhaustion
• Slow startup of background workers
• External API calls that block

A service with a 3-second timeout may fail during startup if it takes 4 seconds to initialize caches.

Probe Body and Response Validation

Health checks are not just HTTP status codes. The response body can be validated.

Example: The readiness probe expects a JSON body with status: "ready", and the liveness probe requires status: "alive".

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  successCondition: "status == 'ready'"
  failureCondition: "status != 'ready'"

This allows for more precise validation than just 200 OK.

Advanced Patterns

Health Check with Status Codes

Use different HTTP status codes to communicate the health state and severity.

Recommended mapping:

• 200 OK: service is healthy
• 206 Partial Content: service is ready but degraded (e.g., cache miss, slow DB)
• 503 Service Unavailable: service not ready (e.g., not bootstrapped, dependencies missing)
• 504 Gateway Timeout: service is alive but slow
• 500 Internal Server Error: critical failure (e.g., unhandled exception in health check)

Clients can act on these codes without parsing the body.

Health Check with Metadata

Include metadata in the health response to help operators and monitoring systems.

{
  "status": "ready",
  "timestamp": 1712345678,
  "version": "v1.4.2",
  "commit": "a1b2c3d4e5f6",
  "environment": "production",
  "uptime_seconds": 12345,
  "memory_used_mb": 150,
  "memory_total_mb": 512,
  "queue_depth": 42,
  "worker_count": 8,
  "dependencies": {
    "db": {
      "status": "healthy",
      "latency_ms": 15,
      "query_count": 1024,
      "error": null
    },
    "cache": {
      "status": "healthy",
      "latency_ms": 8,
      "hits": 923,
      "misses": 78,
      "error": null
    }
  }
}

This data can be used for:

• Alerting on specific thresholds (e.g., queue_depth > 100)
• Auto-scaling based on health metrics
• Debugging startup issues
• Logging health state for audit trails

Health Check with Diagnostic Endpoint

A /health/diagnostics endpoint provides detailed information for debugging.

GET /health/diagnostics HTTP/1.1
Host: api.example.com
Accept: application/json

HTTP/1.1 200 OK
Content-Type: application/json

{
  "system": {
    "hostname": "api-1.example.com",
    "pid": 1234,
    "uptime": "1d 3h 22m",
    "memory": {
      "heap": {
        "used": 150,
        "total": 512,
        "max": 1024
      },
      "non_heap": {
        "used": 45,
        "total": 128
      }
    },
    "jvm": {
      "gc": {
        "count": 142,
        "time_ms": 3045
      },
      "threads": {
        "total": 24,
        "running": 18,
        "blocked": 2
      }
    }
  },
  "services": {
    "database": {
      "connection_pool": {
        "active": 10,
        "idle": 2,
        "total": 12,
        "max": 16
      },
      "transaction_count": 12345,
      "last_query": "SELECT * FROM users WHERE id = $1"
    },
    "cache": {
      "hits": 10234,
      "misses": 876,
      "evictions": 154
    }
  }
}

This endpoint can be used for:

• Debugging performance issues
• Investigating GC pauses
• Validating configuration
• Auditing service state during incident response

Common Pitfalls and Silent Failures

Health Check Fails but Service Is Alive

A service returns 200 OK but fails the health check due to a slow dependency. The orchestration system marks it as unhealthy, but the service continues to serve requests.

Root cause: The health check runs on a separate thread or process that is not synchronized with request handling.

Solution:

• Use a dedicated health check worker that runs independently.
• Include timestamp in health response and compare to system clock.
• Log health check results per probe.

Health Check Timeout, but Service Is Still Running

The health check times out due to a 5-second delay, but the service completes the request after the timeout.

Root cause: The service does not send the response until after the timeout, but the probe waits only for the response body.

Solution:

• Use timeoutSeconds that account for expected processing time.
• Set initialDelaySeconds to be at least the expected startup time.
• Consider using readinessProbe with successThreshold to smooth out transient failures.

Misconfigured Failure Threshold

A service with failureThreshold: 2 fails after two consecutive failures, but the probe interval is 10 seconds.

Effect: A 15-second downtime causes the service to be marked unhealthy after 20 seconds, but traffic is already degraded.

Solution:

• Set failureThreshold to ceil(average_downtime / probe_interval).
• Use successThreshold to avoid flapping.

Health Check Body Not Validated

The service returns 200 OK but the body is not parsed correctly.

Example: The readiness probe expects {"status": "ready"}, but the service returns {"status": "Ready"} (capital R).

Effect: The probe fails due to case mismatch, even though the response is valid.

Solution:

• Define exact expectations for the response body.
• Validate the body in the probe configuration.
• Log the received body on failure.

Health Check Fails, But Dependencies Are Healthy

The /health endpoint returns 503 but all dependencies are healthy.

Root cause: The service itself is not ready — it may be in a state where it can accept requests but cannot process them.

Solution:

• Add a health_check field to the service status.
• Include a startup_phase (e.g., "bootstrapping", "caching", "indexing").
• Use 503 with a Retry-After header to guide clients.

Summary

A health check is not just a ping. It is a signal of readiness, liveness, and dependency health. To be effective, it must be:

• Lightweight: minimal overhead, fast response
• Structured: machine-readable, with clear status codes and metadata
• Configurable: tuned to the service’s startup and failure patterns
• Comprehensive: validating all critical dependencies
• Robust: handling timeouts, failures, and transient conditions

Use /health/ready, /health/live, and /health endpoints consistently. Validate the response body. Set initialDelaySeconds, periodSeconds, timeoutSeconds, and failureThreshold with care. Add /health/diagnostics for deep debugging. Let health checks be the first line of defense in backend reliability.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.