How to Implement Observability: Traces, Metrics, and Logs at Scale

This reference explains how to implement observability across traces, metrics, and logs in systems that serve real users at scale. It focuses on concrete mechanisms, not abstract concepts, and covers the specific configuration, data flow, and failure modes you must understand when deploying observability tools. This matters when systems begin to serve traffic beyond local development, especially if they are distributed or stateful.

Tracing: Follow Requests Through Services

Enable Trace Propagation in HTTP Requests

Trace propagation requires a consistent header format across services. Use traceparent as the standard header name, following the W3C Trace Context specification.

Example trace context header:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

In Go, enable trace propagation with go.opentelemetry.io/otel/propagation:

import "go.opentelemetry.io/otel/propagation"

var propagator = propagation.NewCompositeTextMapPropagator(
    propagation.TraceContext{},
    propagation.Baggage{},
)

Set Up Trace Sampling

Traces are expensive to store. Implement sampling in your tracing agent or library.

Example configuration for OpenTelemetry Collector:

receivers:
  otlp:
    protocols:
      grpc:
exporters:
  jaeger:
    endpoint: "jaeger-collector:14250"
processors:
  batch:
  memory_limiter:
    limit_mib: 100
    spike_limit_mib: 50
    check_interval: 1s
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [jaeger]

Trace IDs Are Not Unique

Trace IDs are not globally unique across services. They may be generated locally and must be correlated by a shared root ID.

Failure mode: A service logs trace_id=abc123 while another service logs trace_id=abc123 for different requests, causing false correlation.

Metrics: Measure System Health

Expose Metrics via HTTP Endpoints

Metrics are often exposed at /metrics and scraped by Prometheus or similar systems.

Example in Python using prometheus_client:

from prometheus_client import Counter, Histogram, start_http_server

request_count = Counter('http_requests_total', 'Total HTTP requests', ['method', 'endpoint'])
request_duration = Histogram('http_request_duration_seconds', 'HTTP request duration')

@app.route('/api/users')
def get_users():
    with request_duration.time():
        # Your logic here
        return jsonify({"users": []})

Use Appropriate Metric Types

Choose between counters, gauges, histograms, and summaries based on use case.

Counter for cumulative counts:

rate(http_requests_total[5m])

Gauge for instantaneous values:

go_memstats_alloc_bytes

Histogram for request duration distribution:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Metric Labels Can Cause Cardinality Issues

Too many unique label combinations cause performance degradation and storage bloat.

Example of problematic metrics:

http_requests_total{user_id="12345", session_id="abcde", request_id="xyz"}

Each unique combination creates a new time series. Avoid labels with high cardinality, such as user IDs or UUIDs, unless strictly necessary.

Logs: Correlate Events and Errors

Standardize Log Format

Use structured logging with consistent keys. JSON is the preferred format for machine processing.

Example log line:

{
  "timestamp": "2023-10-10T12:34:56Z",
  "level": "error",
  "message": "Database connection failed",
  "service": "user-service",
  "trace_id": "0af7651916cd43dd8448eb211c80319c",
  "span_id": "b7ad6b7169203331"
}

Log to Standard Output

For containerized environments, log to stdout/stderr rather than files.

Example Dockerfile:

FROM alpine:latest
COPY app /app
CMD ["/app"]

The container runtime captures stdout and stderr automatically. Logging to files inside containers is not scalable.

Log Rotation and Retention

Set up log rotation on the host or in the container environment.

Example systemd service configuration:

[Service]
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

Logs are rotated by systemd-journald by default, using configured limits.

Integrating Traces, Metrics, and Logs

Correlate Across Systems

Use trace IDs to correlate logs, metrics, and traces in a single request flow.

Example: When a request fails, the error log entry includes trace_id so a developer can search for all related spans and metrics from that trace.

Failure mode: A system has separate trace ID generation and does not propagate it through the entire call chain. Logs may contain unrelated trace IDs, making correlation impossible.

Set Up Centralized Log Collection

Use tools like Fluentd or Vector to collect logs from services and ship them to a central store.

Example Vector configuration:

[sources.in]
type = "stdin"

[transforms.add_metadata]
type = "remap"
inputs = ["in"]
source = '''
  .timestamp = now()
  .service = "user-service"
'''

[sinks.out]
type = "elasticsearch"
inputs = ["add_metadata"]
endpoint = "http://elasticsearch:9200"
index = "logs-{date}"

Avoid Log Bloat

Logs should be concise and include only essential information.

Failure mode: Logging full request bodies or responses for every API call causes disk space issues and log processing overhead.

Configuration and Deployment Patterns

Use Environment Variables for Configuration

Configure observability components using environment variables.

Example:

export OTEL_SERVICE_NAME="user-service"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4317"
export OTEL_TRACES_SAMPLER="parentbased_traceidratio"

Deploy Observability Components Separately

Deploy tracing, metrics, and logging agents as independent services.

Example Kubernetes deployment for OpenTelemetry Collector:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: collector
        image: otel/opentelemetry-collector:0.87.0
        args: ["--config=/etc/otel-collector-config.yaml"]

Enable Health Checks

Ensure that observability components expose liveness and readiness endpoints.

Example health check in Go:

http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
})

Common Failure Modes and Gotchas

Traces Are Not Being Collected

Check if the tracing agent is receiving data. A missing traceparent header or misconfigured exporter can silently drop traces.

Metrics Are Not Scraped

Prometheus requires a /metrics endpoint that returns valid Prometheus format. If it returns HTML or an error, scraping fails.

Example of a broken metrics endpoint:

curl http://localhost:8080/metrics
# Returns HTML instead of text/plain

Logs Are Not Being Parsed

Log parsing in tools like Elasticsearch or Loki depends on consistent log formats and patterns. A change in structure breaks existing queries.

Resource Limits Cause Tracing Drops

If memory or CPU limits are too tight, tracing libraries may drop spans silently. Monitor otel_exporter_otlp_traces_dropped_spans metrics.

Metrics Labels Are Too High Cardinality

A label like user_id with 10 million unique values can cause Prometheus to use excessive memory. Use histogram_quantile with appropriate buckets to avoid this.

Logs Are Not Correlated by Trace ID

If logs do not include trace context, debugging becomes impossible. Verify that every log line includes the current trace and span IDs.

Monitoring Observability Itself

Watch for Collector Failures

Set up alerts on metrics like:

rate(otel_exporter_otlp_traces_dropped_spans[5m]) > 0

This detects when spans are being dropped due to overload or misconfiguration.

Monitor Resource Usage

Watch memory and CPU usage of observability agents.

Example alert:

rate(container_cpu_usage_seconds_total{container="otel-collector"}[5m]) > 0.8

Summary

Observability requires coordination between tracing, metrics, and logs. Use consistent headers for trace propagation, structured logging, and appropriate metric types. Avoid high-cardinality labels, ensure components are properly configured, and monitor the observability stack itself. The key is not just collecting data, but making it usable in production.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.