Platform Observability

Production engineering guide for platform observability covering patterns, implementation strategies, and operational best practices.

Platform Observability is the practice of understanding a platform’s health, performance, and behavior through continuous data collection, correlation, and analysis. It matters when a platform spans multiple services, environments, and teams — when a deployment breaks a system but no one knows which component failed, or when latency spikes appear without clear cause. Observability is not monitoring; it is the ability to answer *why* a system behaves the way it does, not just *that* it does.

Understanding System Behavior

The primary goal of platform observability is to answer the question: *What is happening in the platform, and why?* This requires three pillars: logs, metrics, and traces, collected from every service, infrastructure component, and deployment.

Logs are the raw record of events. Each service emits structured logs using JSON or key-value formats. A typical log entry from a user service might look like:

{
  "timestamp": "2024-08-17T12:34:56.789Z",
  "level": "info",
  "service": "user-service",
  "message": "User created",
  "user_id": "u-12345",
  "request_id": "req-abc123",
  "duration_ms": 124,
  "db_queries": 3
}

Logs are collected via agents like Fluentd or Loki, which run as sidecars or system daemons. Fluentd configures input sources and output sinks with explicit match and source blocks:

<source>
  @type tail
  path /var/log/app/*.log
  pos_file /var/log/fluentd/app.log.pos
  tag app.*
</source>

<match app.*>
  @type forward
  <server>
    host 10.0.1.10
    port 24224
  </server>
</match>

Logs alone are insufficient. Metrics provide aggregated, time-series data — counts, averages, percentiles, and rates. A platform uses Prometheus to scrape metrics from services via HTTP endpoints. Each service exposes a /metrics endpoint with lines in the OpenMetrics format:


# TYPE http_requests_total counter
http_requests_total{method="POST",handler="/api/users"} 1432
http_requests_total{method="GET",handler="/api/users"} 4567
http_request_duration_seconds{method="GET",handler="/api/users",status="200"} 0.123

Prometheus scrapes these endpoints every 30 seconds by default. A prometheus.yml configuration defines the scrape jobs:

scrape_configs:
  - job_name: 'user-service'
    static_configs:
      - targets: ['user-service:8080']
    metrics_path: '/metrics'
    scheme: 'http'
    scrape_interval: 30s

Traces connect individual requests across services, forming a distributed call graph. Each trace consists of spans — units of work with start, end, and metadata. A trace for a user creation request might include spans in the user service, auth service, and database:

Trace ID: req-abc123
├── Span: user-service.create_user (124ms)
│   ├── Start: 2024-08-17T12:34:56.789Z
│   ├── End: 2024-08-17T12:34:56.913Z
│   ├── Tags: user_id=u-12345
│   └── Logs: "Validating input", "Saving to DB"
├── Span: auth-service.authenticate (45ms)
│   ├── Start: 2024-08-17T12:34:56.820Z
│   ├── End: 2024-08-17T12:34:56.865Z
│   └── Tags: token_ttl=3600
└── Span: postgresql.query (28ms)
    ├── Start: 2024-08-17T12:34:56.835Z
    ├── End: 2024-08-17T12:34:56.863Z
    └── Tags: query="INSERT INTO users ..."

Traces are propagated using HTTP headers. When a service makes an HTTP call, it sets:

The traceparent header contains the trace ID, parent span ID, and flags. The tracestate holds vendor-specific or service-specific metadata.

Diagnosing Performance Issues

When a service is slow, the first step is to correlate metrics, logs, and traces.

A dashboard in Grafana shows a spike in http_request_duration_seconds for the /api/users endpoint. The user clicks on a time window and sees:

The next step is to investigate the trace for a slow request. The trace reveals that the auth-service.authenticate span took 45ms, but the postgresql.query span took 28ms — both within expected bounds. However, the user-service.create_user span shows a large gap between start and end: 124ms, but the sum of its internal spans is only 80ms. The remaining 44ms is unaccounted for.

This indicates a bottleneck. The next step is to check the logs for the user service during that period. The logs show:

2024-08-17T12:34:56.789Z [INFO]  User created
2024-08-17T12:34:56.810Z [DEBUG]  Validating input
2024-08-17T12:34:56.820Z [DEBUG]  Saving to DB
2024-08-17T12:34:56.835Z [ERROR]  Failed to send email notification: timeout
2024-08-17T12:34:56.913Z [INFO]  User created

The error message is critical: *Failed to send email notification: timeout*. The user service calls an email service asynchronously, but the call times out after 100ms. The trace shows no span for the email service — the call was made, but the trace was not propagated.

This leads to a common failure: missing trace propagation. The service calls the email service using curl or HTTP client, but fails to set the traceparent header. The email service logs the request, but the trace is not connected.

To fix this, the user service must extract the current trace context and inject it into outgoing HTTP calls:

func createUser(ctx context.Context, user User) error {
    span := otel.Tracer("user-service").Start(ctx, "create_user")
    defer span.End()

    // Validate input
    if err := validateUser(user); err != nil {
        span.AddEvent("validation_failed", trace.WithAttributes(attribute.String("error", err.Error())))
        return err
    }

    // Save to DB
    dbCtx := context.WithValue(ctx, "db", "users")
    if err := db.Save(user, dbCtx); err != nil {
        span.AddEvent("db_save_failed", trace.WithAttributes(attribute.String("error", err.Error())))
        return err
    }

    // Send email notification
    emailCtx := otel.Propagator().Extract(ctx, propagation.HTTPTextMapCarrier{
        Get: func(key string) string {
            return http.Header{key: []string{""}}[key]
        },
        Set: func(key, value string) {
            http.Header[key] = []string{value}
        },
    })

    req, _ := http.NewRequestWithContext(emailCtx, "POST", "http://email-service/send", nil)
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("traceparent", otel.Propagator().Inject(ctx, propagation.HTTPTextMapCarrier{}).Get("traceparent"))

    client := &http.Client{Timeout: 100 * time.Millisecond}
    _, err := client.Do(req)
    if err != nil {
        span.AddEvent("email_timeout", trace.WithAttributes(attribute.String("error", err.Error())))
        return err
    }

    return nil
}

This code ensures that the trace context is passed across the service boundary. The email service now logs the request with the correct traceparent, and the full trace becomes visible.

Correlating Events Across Systems

A common failure mode is time drift. Services run on different machines, possibly in different time zones or with misaligned clocks. A log entry from the user service says "User created at 12:34:56.789", but the database logs the same event at 12:34:57.123. The difference is 334ms — not a rounding error.

To correlate logs, metrics, and traces, the platform must align timestamps using UTC and NTP. All services must be synchronized to a common time source. The timedatectl command on Linux systems shows:

timedatectl status
# Output:
#               Local time: Mon 2024-08-17 12:34:56 UTC
#          Universal time: Mon 2024-08-17 12:34:56 UTC
#                RTC time: Mon 2024-08-17 12:34:56
#               Time zone: UTC (UTC, +0000)
# System clock synchronized: yes
#              RTC in local TZ: no

Even with NTP, the clock may drift. A service may start with a timestamp that is 100ms behind the system clock. The platform uses journalctl to collect logs from systemd services:

journalctl --since "2024-08-17 12:34:00" --until "2024-08-17 12:35:00" \
           --unit user-service \
           --output json \
           --follow

The --output json flag ensures structured logs. The --follow flag streams new logs in real time. The output is consumed by a log aggregation system like Loki, which stores logs in a time-series database.

Loki uses labels to group logs. A label is a key-value pair attached to a stream of logs. The user service logs are labeled with service=user-service, instance=us-1, and job=user-service. Queries in Loki use a PromQL-like syntax:

{service="user-service", job="user-service"} |= "email_timeout"

This query returns all log entries from the user service where the log message contains “email_timeout”. The result is a list of entries with full context — timestamps, log level, and structured fields.

Detecting and Alerting on Anomalies

The platform uses alerts to detect anomalies. Alerts are defined using rules in Prometheus. A rule file defines conditions that trigger alerts when thresholds are crossed.

groups:
  - name: user-service
    interval: 5m
    rules:
      - alert: HighRequestLatency
        expr: |
          histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
          > 0.5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "95th percentile request latency > 500ms"
          description: "The 95th percentile latency for user-service requests has been above 500ms for the last 10 minutes."

This rule checks if the 95th percentile of request duration over the last 5 minutes exceeds 500ms. If so, it triggers an alert after 10 minutes of sustained high latency.

Alerts are evaluated every 5 minutes. The alerting engine uses alertmanager to send notifications. Alertmanager supports multiple notification methods: email, Slack, PagerDuty, and webhooks.

The configuration for Alertmanager is a single YAML file:

global:
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: '[email protected]'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'secret'

route:
  receiver: 'team-ops'
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h

receivers:
  - name: 'team-ops'
    email_configs:
      - to: '[email protected]'
        send_resolved: true
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/...'
        channel: '#platform-alerts'
        title: '{{ template "slack.title" . }}'
        text: '{{ template "slack.text" . }}'

When an alert fires, the platform sends a Slack message with the alert title and body. The title and text are rendered using Go templates, with access to all alert metadata.

Managing Observability Data at Scale

As the platform grows, the volume of observability data increases. The platform must manage data retention, compression, and query performance.

Loki uses index and chunk storage. Each log stream is stored as a sequence of chunks, with an index that maps labels to chunk locations. The index is stored in a distributed database like Cassandra or Bigtable.

Prometheus uses remote storage to offload old metrics. The configuration specifies a remote write endpoint:

remote_write:
  - url: 'http://prometheus-remote-storage:8080/write'
    queue_config:
      max_samples_per_send: 1000
      max_shards: 100
      capacity: 10000
      batch_timeout: 30s
    resource:
      labels:
        job: 'user-service'
      resource: 'user-service'

This config sends batches of samples to a remote storage service every 30 seconds, up to 1000 samples per batch. The remote storage service must accept the OpenMetrics format over HTTP.

The platform also uses log rotation and compression. Log files are rotated daily using logrotate:

/var/log/app/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
        systemctl reload fluentd
    endscript
}

This ensures that old logs are compressed and archived, reducing storage costs.

Common Pitfalls and Sharp Edges

The platform must instrument services with health checks and liveness probes. A service exposes a /health endpoint that returns:

{
  "status": "healthy",
  "checks": [
    {
      "name": "database",
      "status": "ok",
      "details": {
        "latency_ms": 12,
        "connected": true
      }
    },
    {
      "name": "cache",
      "status": "warning",
      "details": {
        "latency_ms": 45,
        "connected": true,
        "miss_rate": 0.23
      }
    }
  ]
}

Kubernetes uses this to determine if a pod is ready. The readinessProbe and livenessProbe in a Pod manifest use the /health endpoint to decide when to start traffic or restart a container.

Summary

Platform observability is not a collection of tools, but a practice of understanding systems through data. It requires consistent logging, metric collection, and distributed tracing. The key is to correlate data across services and time. The sharp edges — missing trace propagation, time drift, batching delays — are the places where systems fail silently. The platform must be instrumented to detect and diagnose issues before they impact users. The mechanism — not the tool — defines the practice.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.