Platform Reliability | The Garnet Wiki

Platform Reliability is the discipline of ensuring that a platform consistently delivers its services with predictable performance, minimal downtime, and rapid recovery from failures. It matters most when platform components are shared across teams, services are composed from multiple platform-backed capabilities, and the cost of a failure scales beyond a single service. Platform reliability is not just monitoring and alerting—it is the practice of designing, measuring, and improving the resilience of the platform infrastructure and its services from the ground up.

Designing for Observability and Diagnostics

Observability is the foundation of platform reliability. Without it, failures are invisible until they impact users, and debugging is a guessing game. The core observability stack consists of logs, metrics, and distributed traces, each with distinct formats and workflows.

Logs are structured events emitted by services. They are written to files, streamed via standard output, and collected by agents. Each log line follows a consistent schema: a timestamp, a severity level, a source context (service name, instance ID), and a structured payload. A typical log entry in JSON format:

{
  "timestamp": "2023-10-05T12:34:56.789Z",
  "level": "error",
  "service": "user-service",
  "instance": "user-service-7f4d3c2b1a",
  "message": "Failed to create user: invalid email format",
  "error": {
    "type": "ValidationError",
    "code": "EMAIL_INVALID",
    "details": [
      "email must be a valid RFC 822 address",
      "email length must be between 5 and 254 characters"
    ]
  },
  "requestId": "req-abc123def456"
}

Metrics are numerical values collected at regular intervals. They are reported via the OpenTelemetry protocol or a simple HTTP POST to a metrics endpoint. A common metric format uses a time-series schema:

PUT /metrics
Content-Type: application/json

[
  {
    "name": "http.request.duration",
    "value": 342.5,
    "labels": {
      "service": "order-service",
      "method": "POST",
      "status": "201"
    },
    "timestamp": "2023-10-05T12:34:56.789Z"
  },
  {
    "name": "disk.usage",
    "value": 0.78,
    "labels": {
      "host": "app-server-01",
      "mount": "/var/lib/data"
    },
    "timestamp": "2023-10-05T12:34:56.789Z"
  }
]

Traces link requests across services. Each trace is a tree of spans, where each span represents a unit of work. Spans are propagated via HTTP headers: traceparent, tracestate, and baggage. The traceparent header follows the W3C Trace Context specification:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e5686-00f060a5875d7265-01

A service receiving a request adds its own span and forwards the header to downstream calls. The root span is created by the entry point (e.g., API gateway). Spans are recorded in a structured format:

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e5686",
  "span_id": "00f060a5875d7265",
  "parent_id": null,
  "name": "POST /users",
  "start_time": "2023-10-05T12:34:56.789Z",
  "end_time": "2023-10-05T12:34:57.131Z",
  "attributes": {
    "http.method": "POST",
    "http.url": "/users",
    "http.status_code": 201,
    "user.id": 12345
  },
  "events": [
    {
      "name": "db.query",
      "timestamp": "2023-10-05T12:34:56.920Z",
      "attributes": {
        "sql": "INSERT INTO users (name, email) VALUES ($1, $2)",
        "rows_affected": 1
      }
    }
  ]
}

Setting Up Automated Health Checks and Probes

Health checks are essential for detecting when a service is alive and ready to serve requests. Platforms use two types: liveness and readiness probes.

A liveness probe determines whether a service is running. If the probe fails, the platform restarts the service. A readiness probe determines whether a service is ready to accept traffic. A service is added to the load balancer only when all readiness probes pass.

Health checks are defined in a service’s configuration file. For Kubernetes, this is part of the Pod specification:

apiVersion: v1
kind: Pod
metadata:
  name: user-service
spec:
  containers:
    - name: user-service
      image: user-service:v2.1
      ports:
        - containerPort: 8080
      livenessProbe:
        httpGet:
          path: /health/liveness
          port: 8080
        initialDelaySeconds: 30
        periodSeconds: 10
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /health/readiness
          port: 8080
        initialDelaySeconds: 10
        periodSeconds: 5
        failureThreshold: 2

The /health/liveness endpoint returns a 200 status code and a JSON body:

{
  "status": "UP",
  "timestamp": "2023-10-05T12:34:56.789Z",
  "checks": [
    {
      "name": "database",
      "status": "UP",
      "details": {
        "connection": "OK",
        "latency_ms": 12
      }
    },
    {
      "name": "cache",
      "status": "UP",
      "details": {
        "connected": true,
        "key_count": 4567
      }
    }
  ]
}

The /health/readiness endpoint includes additional checks for startup dependencies:

{
  "status": "READY",
  "timestamp": "2023-10-05T12:34:56.789Z",
  "checks": [
    {
      "name": "database",
      "status": "READY",
      "details": {
        "connection": "OK",
        "latency_ms": 8,
        "migrations_completed": true
      }
    },
    {
      "name": "message_queue",
      "status": "READY",
      "details": {
        "consumers_started": true,
        "lag_seconds": 0.7
      }
    }
  ]
}

A common failure mode: a service starts but takes longer than expected to initialize. The readiness probe fails initially, so the service is not added to the load balancer. After 10 seconds, the readiness probe passes, and traffic begins. But if the database connection drops after the service starts, the readiness check fails—traffic continues, but the service is now unresponsive.

Automating Recovery with Self-Healing Mechanisms

Self-healing is the practice of automatically restoring service health without human intervention. It relies on three mechanisms: automated restarts, rollbacks, and scaling.

When a liveness probe fails, the platform restarts the container. The platform logs the restart event, and the new container instance begins serving requests. The restart is triggered by a health check failure, but the platform also monitors the number of restarts. After three consecutive failures, the platform marks the service as unstable and triggers a rollback.

Rollbacks are initiated when a new version of a service fails to meet health criteria. A platform compares the current version against a baseline. A health check threshold is defined: e.g., error rate must be below 1% and latency below 200ms. If a new version fails this threshold for five consecutive minutes, the platform initiates a rollback.

The rollback process is orchestrated by a control plane. It sends a command to the deployment system:

curl -X POST https://platform-api.example.com/deployments/rollback \
  -H "Content-Type: application/json" \
  -d '{
    "deploymentId": "dep-abc123",
    "reason": "High error rate in v2.2",
    "triggeredBy": "health-check",
    "targetVersion": "v2.1"
  }'

The deployment system then updates the service’s desired state. For Kubernetes, this is a Deployment object:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service
spec:
  replicas: 4
  selector:
    matchLabels:
      app: user-service
  template:
    metadata:
      labels:
        app: user-service
        version: v2.1
    spec:
      containers:
        - name: user-service
          image: user-service:v2.1
          ports:
            - containerPort: 8080

Scaling is driven by metrics. The platform monitors key performance indicators and adjusts the number of replicas. A horizontal pod autoscaler (HPA) is configured with a target metric:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: user-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: user-service
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Pods
      pods:
        metric:
          name: http.request.count
          target:
            type: AverageValue
            averageValue: 100
        windowSeconds: 300

When CPU utilization exceeds 70%, or the average request count per pod exceeds 100, the platform increases the number of replicas. The platform logs the scaling events:

{
  "event": "scale_up",
  "timestamp": "2023-10-05T12:34:56.789Z",
  "deployment": "user-service",
  "from": 4,
  "to": 8,
  "reason": "CPU utilization above threshold",
  "metrics": {
    "cpu": 76.2,
    "request_count": 123
  }
}

A sharp edge: scaling based on request count is sensitive to request size and distribution. A spike in large requests may cause scaling, but the platform may not react to a spike in small requests unless the metric is tuned. Also, scaling up is not instantaneous. A new replica is scheduled, pulled, started, and waits for readiness probes. During this time, the service may experience cold starts, and requests are not yet distributed evenly.

Managing Failure States and Outages

When a service fails, the platform must surface the failure, prioritize it, and guide recovery. Failures are captured in alerts, which are triggered by threshold violations.

An alert is defined by a query over metrics or logs. For example, a query for high error rates:

SELECT
  service,
  COUNT(*) AS error_count,
  AVG(latency_ms) AS avg_latency,
  PERCENTILE(latency_ms, 0.95) AS p95_latency
FROM logs
WHERE level = 'error'
  AND timestamp >= NOW() - INTERVAL '5 minutes'
GROUP BY service
HAVING COUNT(*) > 100 OR AVG(latency_ms) > 500

Alerts are sent via email, Slack, or a PagerDuty integration. A typical alert payload includes:

{
  "alertId": "alert-12345",
  "title": "High error rate in user-service",
  "summary": "Error rate in user-service exceeded 5% over 5 minutes",
  "timestamp": "2023-10-05T12:34:56.789Z",
  "details": {
    "service": "user-service",
    "duration": "5m",
    "error_rate": 6.3,
    "threshold": 5.0,
    "top_errors": [
      {
        "message": "Failed to create user: invalid email format",
        "count": 87,
        "example": {
          "requestId": "req-abc123def456",
          "email": "john@doe",
          "error": {
            "type": "ValidationError",
            "code": "EMAIL_INVALID"
          }
        }
      }
    ]
  }
}

When alerts are triggered, the platform creates incident tickets. The platform also marks the service as degraded and updates the platform’s status page. A status page entry includes:

{
  "component": "user-service",
  "status": "degraded",
  "last_updated": "2023-10-05T12:34:56.789Z",
  "description": "High error rate and increased latency",
  "incidents": [
    {
      "id": "inc-abc123",
      "title": "Email validation failure in user creation",
      "started_at": "2023-10-05T12:30:00.000Z",
      "status": "triaged",
      "details": [
        "Error rate: 6.3%",
        "P95 latency: 720ms",
        "Top error: invalid email format"
      ]
    }
  ],
  "metrics": [
    {
      "name": "error_rate",
      "value": 6.3,
      "trend": "rising",
      "last_updated": "2023-10-05T12:34:56.789Z"
    },
    {
      "name": "p95_latency",
      "value": 720,
      "trend": "rising",
      "last_updated": "2023-10-05T12:34:56.789Z"
    }
  ]
}

Failure states are managed through a state machine. A service can be in one of several states: initializing, healthy, degraded, unavailable, maintenance. The platform transitions between states based on health check results, alert thresholds, and manual actions.

Ensuring Consistency and Predictability

Platform reliability is not just reactive—it is proactive. A platform ensures consistency by defining and enforcing golden paths: standardized, well-documented ways to deploy, monitor, and operate services.

A golden path for a service includes:

A standard Dockerfile with specific build stages and entrypoints.
A kubernetes.yaml file with default resource requests and limits.
A prometheus.yml configuration with default scrape jobs.
A logging.yaml configuration that emits structured logs with consistent schema.
A health-check.yaml file that defines both liveness and readiness probes.

The platform validates that services conform to the golden path. It runs a validation job on every deployment:

docker run --rm \
  -v /path/to/deployment:/app \
  -e PLATFORM_ENV=production \
  quay.io/platform-validator:v1.2 \
  validate \
    --config /app/health-check.yaml \
    --schema /app/logging.schema.json \
    --resources /app/resources.yaml

The validator outputs a report:

{
  "success": false,
  "issues": [
    {
      "type": "missing",
      "path": "spec.containers[0].livenessProbe",
      "message": "Liveness probe is missing"
    },
    {
      "type": "mismatch",
      "path": "spec.containers[0].image",
      "expected": "user-service:v2.1",
      "actual": "user-service:v2.2",
      "message": "Image version does not match golden path"
    },
    {
      "type": "warning",
      "path": "spec.containers[0].resources.limits.cpu",
      "message": "CPU limit is lower than recommended"
    }
  ],
  "summary": "3 issues found, 1 critical, 2 warnings"
}

The platform blocks deployments that fail validation. A deployment is marked as failed if the validation report includes any critical issues.

The platform also enforces versioning. Every service has a version label in its metadata. When a service is deployed, the platform updates a central service-registry table:

INSERT INTO service_registry (
  name,
  version,
  image,
  environment,
  deployed_at,
  health_status
) VALUES (
  'user-service',
  'v2.2',
  'user-service:v2.2',
  'production',
  '2023-10-05T12:34:56.789Z',
  'healthy'
);

This registry is used for service discovery, dependency tracking, and rollbacks. When a service fails, the platform queries the registry to find all versions and their health status.

Platform reliability is the practice of building a platform that does not just run—but runs well, recovers from failure, and improves over time. It is the difference between a platform that is merely available and one that is trusted.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.