Observability Pipeline

Production engineering guide for observability pipeline covering patterns, implementation strategies, and operational best practices.

An observability pipeline transforms raw telemetry—logs, metrics, traces, and events—into actionable insights across services, infrastructure, and applications. It matters when a system’s health, performance, or correctness cannot be inferred from isolated tools or dashboards. It is essential when debugging latency spikes, detecting subtle regressions, or correlating incidents across microservices. The pipeline integrates data from sources into a unified, persistent store, where it is enriched, processed, and served to users through alerting, visualization, and search.

Data Ingestion

The first stage of the pipeline collects telemetry from services and infrastructure. Data arrives via protocols like gRPC, HTTP, or UDP, and must be reliably delivered even under network instability or high load.

Log Ingestion

Logs are ingested using structured formats—JSON, Protobuf, or plain text—with a common schema across services. Each log entry includes timestamp, level, message, and trace_id. A service sends logs using curl to a central collector:

curl -X POST http://collector.example.com/logs \
  -H "Content-Type: application/json" \
  -d '{
    "timestamp": "2023-09-15T12:34:56.789Z",
    "level": "ERROR",
    "message": "Failed to connect to database",
    "trace_id": "a1b2c3d4e5f6",
    "service": "user-service",
    "component": "db-client",
    "error": {
      "type": "ConnectionTimeout",
      "code": 1001,
      "details": "Connection timed out after 5s"
    }
  }'

The collector validates the payload, checks for required fields, and rejects malformed entries. A rejected log is sent to a dead-letter queue with an error code and original body. Common failures include missing timestamp, incorrect level values (INFO vs Info), and trace_id format mismatches.

Metric Ingestion

Metrics are submitted as batches of time-series data, with name, value, timestamp, and labels. The OpenTelemetry Protocol (OTLP) is used, but the pipeline accepts any JSON-based metric format. A service sends metrics via curl:

curl -X POST http://collector.example.com/metrics \
  -H "Content-Type: application/json" \
  -d '[
    {
      "name": "http.request.duration",
      "value": 234.7,
      "timestamp": "2023-09-15T12:34:56.789Z",
      "labels": {
        "method": "POST",
        "status": "200",
        "service": "order-api"
      }
    },
    {
      "name": "disk.usage.bytes",
      "value": 1073741824,
      "timestamp": "2023-09-15T12:34:56.789Z",
      "labels": {
        "device": "/data",
        "host": "db-01"
      }
    }
  ]'

The pipeline ensures that metric names are normalized (e.g., http.request.duration → http_request_duration) and that timestamp is in ISO 8601 with UTC. A common failure mode is misaligned timestamps: a metric recorded at 12:34:56Z arrives at 12:34:57Z in the collector, causing gaps or overlaps in time-series aggregation.

Trace Ingestion

Traces are sent as spans, each with trace_id, span_id, parent_id, name, start_time, end_time, and attributes. The pipeline accepts OTLP, but also supports a legacy JSON format:

[
  {
    "trace_id": "a1b2c3d4e5f6",
    "span_id": "c3d4e5f6a1b2",
    "parent_id": "b2c3d4e5f6a1",
    "name": "handle_order",
    "start_time": "2023-09-15T12:34:56.789Z",
    "end_time": "2023-09-15T12:34:57.123Z",
    "attributes": {
      "request.id": "req-12345",
      "user.id": 1001
    }
  }
]

A trace is validated for completeness: every span must have a trace_id, and parent_id must refer to an existing span in the same trace. A missing parent_id or incorrect trace_id format leads to fragmented traces. The pipeline enriches spans with service and host metadata from a registry.

Data Processing

Once ingested, data is processed to enrich, aggregate, and transform. This stage runs in a stream processing engine—like Apache Kafka, Flink, or a custom pipeline using kafka-topics.sh, kafka-console-consumer.sh, and kafka-console-producer.sh.

Enrichment

Each log, metric, and trace is enriched with metadata: service version, deployment environment, cluster location, and host IP. The pipeline subscribes to a metadata topic, where services publish updates:

{
  "service": "payment-service",
  "version": "v3.2.1",
  "environment": "staging",
  "cluster": "us-west-2",
  "pod": "payment-7b4c9f2d8",
  "ip": "10.0.2.15",
  "labels": {
    "team": "payments",
    "tier": "backend"
  }
}

A processor joins this metadata with incoming logs using service and pod fields. If a log arrives with service: payment-service but pod: payment-7b4c9f2d8, the processor looks up the metadata and adds version, environment, and labels to the log.

A common failure is mismatched pod naming: a service uses payment-7b4c9f2d8 in logs but payment-7b4c9f2d8 in metadata, with a trailing space or capitalization difference. The pipeline uses case-insensitive matching, but a pod field with Payment-7b4c9f2d8 fails to join.

Aggregation

Metrics are aggregated at multiple intervals: 1s, 1m, 5m, 1h. The pipeline uses a windowed processing model, where it groups metrics by name, labels, and a time window. For example:

kafka-console-consumer.sh \
  --bootstrap-server kafka.example.com:9092 \
  --topic metrics-raw \
  --partition 0 \
  --offset latest \
  --max-messages 1000

A processor reads raw metrics, groups them by name, labels, and window, then emits aggregated metrics with sum, mean, count, min, and max values.

A critical failure mode is window boundary drift: a metric arrives at 12:34:56.789Z, but the window is defined as 12:34:00.000Z to 12:35:00.000Z. The processor assigns it to the wrong window if timestamps are not aligned to the nearest second. The pipeline handles this by normalizing timestamps to the start of the window, but a misaligned start_time leads to inaccurate aggregations.

Traces are aggregated into service-to-service call graphs by merging spans with the same trace_id. The pipeline extracts duration and error flags, and builds a DAG of spans. A missing parent_id causes a trace to split into disconnected sub-traces.

Data Storage

Processed data is stored in a durable, queryable format. The pipeline uses a columnar database for metrics, a time-series database for logs, and a graph store for traces.

Metrics Storage

Metrics are stored in a time-series database (e.g., InfluxDB, TimescaleDB, or a custom PostgreSQL schema). The pipeline writes to a table with the following schema:

CREATE TABLE metrics (
  name TEXT NOT NULL,
  labels JSONB NOT NULL,
  timestamp TIMESTAMPTZ NOT NULL,
  value DOUBLE PRECISION NOT NULL,
  count BIGINT,
  mean DOUBLE PRECISION,
  min DOUBLE PRECISION,
  max DOUBLE PRECISION,
  PRIMARY KEY (name, labels, timestamp)
);

Data is written in batches using COPY or INSERT statements. A common failure is timestamp precision mismatch: the pipeline writes TIMESTAMP WITH TIME ZONE values, but the database expects TIMESTAMP WITHOUT TIME ZONE, leading to time shifts.

Logs Storage

Logs are stored in a columnar format (e.g., Apache Parquet or a custom JSONL file) in a distributed file system (e.g., S3, HDFS, or NFS). Each log entry is stored as a row with timestamp, level, message, trace_id, service, component, and error fields.

The pipeline writes logs in batches of 10,000 entries, using a JSONL format:

{"timestamp":"2023-09-15T12:34:56.789Z","level":"ERROR","message":"Failed to connect to database","trace_id":"a1b2c3d4e5f6","service":"user-service","component":"db-client","error":{"type":"ConnectionTimeout","code":1001,"details":"Connection timed out after 5s"}}
{"timestamp":"2023-09-15T12:34:57.123Z","level":"INFO","message":"Database connected","trace_id":"a1b2c3d4e5f6","service":"user-service","component":"db-client","error":null}

A failure occurs when the file system is slow: a batch of logs takes 15 seconds to write, but the next batch arrives while the first is being written. The pipeline uses a two-phase commit: first write to a staging directory, then atomically move to the final location. A failed move leaves logs in an inconsistent state.

Traces Storage

Traces are stored as a graph in a database that supports hierarchical data. The pipeline uses a relational table with trace_id, span_id, parent_id, name, start_time, end_time, and attributes. A trace_events table links spans to events:

CREATE TABLE traces (
  trace_id BYTEA PRIMARY KEY,
  service TEXT,
  duration_ms BIGINT,
  error BOOLEAN,
  root_span_id BYTEA
);

CREATE TABLE spans (
  span_id BYTEA PRIMARY KEY,
  trace_id BYTEA REFERENCES traces(trace_id),
  parent_id BYTEA,
  name TEXT,
  start_time TIMESTAMPTZ,
  end_time TIMESTAMPTZ,
  attributes JSONB
);

CREATE TABLE trace_events (
  trace_id BYTEA,
  span_id BYTEA,
  event_time TIMESTAMPTZ,
  name TEXT,
  attributes JSONB,
  PRIMARY KEY (trace_id, span_id, event_time)
);

A trace is reconstructed by joining spans and trace_events, then ordering by start_time. A failure occurs when parent_id is null but trace_id is not: the root span is missing or improperly linked.

Query and Serving

The final stage makes data available for visualization, alerting, and search.

Query Interface

Clients query the pipeline via HTTP. A query for logs returns a JSON response with results, total, and next_page_token:

{
  "results": [
    {
      "timestamp": "2023-09-15T12:34:56.789Z",
      "level": "ERROR",
      "message": "Failed to connect to database",
      "trace_id": "a1b2c3d4e5f6",
      "service": "user-service",
      "component": "db-client"
    }
  ],
  "total": 1234,
  "next_page_token": "eyJ0aW1lc3RhbXAiOiIyMDIzLTExLTAxVDEyOjM0OjU2Ljc4OVoiLCJsZWZ0IjoiMTIzNCJ9"
}

The next_page_token is a base64-encoded JSON object with timestamp and left count. Clients use this to paginate through large result sets.

Alerting

Alerts are defined as queries over metrics or logs. A rule triggers when a metric exceeds a threshold:

name: "High Error Rate in User Service"
query: |
  SELECT
    mean(error_count) / count(total_requests) AS error_rate
  FROM metrics
  WHERE
    name = 'http.request.error.count'
    AND service = 'user-service'
    AND timestamp > now() - 10m
  GROUP BY
    window(1m)
  HAVING
    error_rate > 0.1
interval: 30s
threshold: 0.1

The pipeline evaluates the query every 30 seconds. When the error_rate exceeds 0.1 for three consecutive evaluations, an alert is created. The alert includes the query result, raw metric values, and a link to a dashboard.

Dashboarding

Dashboards are composed of panels, each a query over metrics or logs. A panel shows the 5-minute average of http.request.duration for order-api:

{
  "title": "Order API Latency",
  "query": "SELECT mean(value) FROM metrics WHERE name = 'http.request.duration' AND service = 'order-api' GROUP BY window(5m)",
  "type": "line",
  "y_axis": "milliseconds"
}

The dashboard engine renders the panels in real time, using cached results and streaming updates.

Failure Modes

The pipeline silently fails when a component becomes unavailable or produces incorrect data. Common failure modes:

The pipeline is resilient, but these silent failures accumulate, making debugging harder and insights less trustworthy.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.