Structured Logging

Production engineering guide for structured logging covering patterns, implementation strategies, and operational best practices.

Structured logging is the practice of emitting application events in a consistent, machine-readable format—typically JSON or key-value pairs—so that logs can be indexed, queried, filtered, and analyzed at scale. It matters most when systems grow beyond single-service monoliths, when teams need to correlate events across services, troubleshoot distributed failures, or monitor health across a fleet of microservices.

Log Format and Encoding

Logs must be serializable, predictable, and self-describing. The most common format is JSON, but key-value pairs (e.g., level=info ts=2023-08-15T12:34:56Z msg="user login success" user_id=42) are equally valid in systems that prioritize simplicity and parsing speed.

JSON Logging

A typical structured log entry in JSON follows a consistent schema:

{
  "timestamp": "2023-08-15T12:34:56.789Z",
  "level": "error",
  "service": "auth-service",
  "trace_id": "a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8",
  "span_id": "x9y8z7-w6v5-u4t3-s2r1",
  "message": "failed to authenticate user",
  "error": {
    "code": "AUTH_401",
    "message": "invalid credentials",
    "stack": "at AuthController.authenticate(AuthController.java:112)\n at UserLoginHandler.handle(UserLoginHandler.java:45)",
    "details": {
      "input": {
        "email": "[email protected]",
        "password": "*****"
      },
      "user_id": 42,
      "request_id": "req-abc123"
    }
  },
  "metadata": {
    "db_duration_ms": 14,
    "cache_hit": true,
    "request_size_bytes": 1024,
    "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"
  }
}

This structure enables efficient querying and alerting. Tools like Elasticsearch, Loki, or even PostgreSQL can index and filter logs by any field.

Key-Value Encoding

For systems with performance constraints or limited parsing capabilities, key-value encoding is preferred:

ts=2023-08-15T12:34:56.789Z level=error service=auth-service trace_id=a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8 msg="failed to authenticate user" error.code=AUTH_401 error.message="invalid credentials" error.details.input.email="[email protected]" error.details.input.password="*****" db_duration_ms=14 cache_hit=true request_size_bytes=1024 user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"

This format is particularly effective when logs are written directly to files or streamed over UDP.

Log Levels and Severity

Log levels are standardized across systems. The five primary levels—debug, info, warn, error, fatal—are universally recognized and expected.

A log entry must include a level field with one of these values. The level field is critical for filtering and alerting.

{
  "level": "warn",
  "message": "high CPU usage in worker pool",
  "metadata": {
    "cpu_percent": 87.3,
    "worker_count": 24,
    "threshold": 80.0
  }
}

The level field is not just metadata—it is the primary axis for log ingestion, routing, and alerting.

Tracing and Context Propagation

In distributed systems, logs must link together across services. The core mechanism is the trace_id and span_id fields.

When a service receives a request with a trace_id, it should propagate the same trace_id to all downstream calls, and optionally assign a span_id to each operation.

{
  "service": "order-service",
  "trace_id": "a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8",
  "span_id": "x9y8z7-w6v5-u4t3-s2r1",
  "message": "processing order #1234",
  "metadata": {
    "order_id": 1234,
    "total_items": 5
  }
}

The trace_id must be passed via HTTP headers. The standard headers are:

These headers are used by all services in the chain.

HTTP Header Propagation

When a service calls another service, it must:

  1. Read X-Trace-ID and X-Span-ID from the incoming request.
  2. Generate a new span_id for the current operation.
  3. Set X-Trace-ID and X-Span-ID in the outgoing request.

Example:

GET /api/v1/users/42 HTTP/1.1
Host: auth-service.example.com
X-Trace-ID: a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8
X-Span-ID: x9y8z7-w6v5-u4t3-s2r1

The auth-service reads these headers, logs them, and forwards them in its own request.

Error Handling and Stack Traces

Errors must be serialized as structured objects. The error field is the canonical location for error details.

Error Object Structure

"error": {
  "code": "AUTH_401",
  "message": "invalid credentials",
  "stack": "at AuthController.authenticate(AuthController.java:112)\n at UserLoginHandler.handle(UserLoginHandler.java:45)",
  "details": {
    "input": {
      "email": "[email protected]",
      "password": "*****"
    },
    "user_id": 42,
    "request_id": "req-abc123"
  }
}

The details field is especially important for debugging. It allows correlation of error logs with request payloads, database queries, and service configurations.

Log Aggregation and Querying

Logs are only valuable when they are collected, indexed, and queryable. The process involves:

  1. Log collection: agents or daemons that read logs from files, stdin, or network streams.
  2. Indexing: storing logs in a searchable backend.
  3. Querying: retrieving logs using filters, aggregations, and time ranges.

Log Collection with Fluentd

Fluentd is a widely used log collector. A typical configuration file (fluentd.conf) for collecting JSON logs from a service:

<source>
  @type forward
  bind 0.0.0.0
  port 24224
</source>

<match **>
  @type file
  path /var/log/fluentd
  format json
  time_key timestamp
  time_format %Y-%m-%dT%H:%M:%S.%LZ
</match>

<match **>
  @type elasticsearch
  host elasticsearch.example.com
  port 9200
  index_name fluentd
  type_name log
</match>

This configuration listens on 0.0.0.0:24224 for incoming logs via Fluentd’s forward protocol, writes them to disk in JSON format, and sends them to Elasticsearch.

The service writes logs directly to stdout in JSON format:

{
  "timestamp": "2023-08-15T12:34:56.789Z",
  "level": "error",
  "message": "failed to authenticate user",
  "error": {
    "code": "AUTH_401",
    "message": "invalid credentials"
  }
}

Fluentd picks up the logs and forwards them to the backend.

Querying Logs in Elasticsearch

A typical query to find all errors in the auth-service over the last hour:

GET /fluentd/_search
{
  "query": {
    "bool": {
      "must": [
        { "term": { "service": "auth-service" } },
        { "range": { "timestamp": { "gte": "now-1h" } } }
      ],
      "filter": [
        { "term": { "level": "error" } }
      ]
    }
  },
  "size": 100,
  "sort": [
    { "timestamp": { "order": "desc" } }
  ]
}

The result is a list of structured logs, each with the full error object and metadata.

Log Rotation and Retention

Logs grow quickly. Without rotation, disk space fills rapidly, and queries become slow.

Log Rotation with logrotate

A logrotate configuration file (/etc/logrotate.d/auth-service):

/var/log/auth-service/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 644 root root
    postrotate
        systemctl reload fluentd
    endscript
}

This rotates logs daily, keeps 30 days of logs, compresses them, and reloads Fluentd after rotation.

Retention Policies

Retention policies define how long logs are kept and at what cost.

The system must define retention policies per service or per log type.

Common Pitfalls and Silent Failures

1. Missing Trace IDs

A service receives a request but fails to propagate X-Trace-ID. Logs from downstream services have no connection to upstream ones. Correlation becomes impossible.

This failure is silent: no error message, no metric spike, but logs are disconnected.

2. Misplaced Error Fields

The error object is nested, but many services place error details directly under the root:

{
  "level": "error",
  "message": "failed to authenticate user",
  "code": "AUTH_401",
  "details": "invalid credentials",
  "stack": "at AuthController.authenticate(...)"
}

This makes error querying difficult. Tools cannot extract code, stack, or details from a single field.

3. Timestamp Format Inconsistency

Logs use different timestamp formats:

The system fails to filter logs by time range, and alerts miss events.

4. Incomplete Metadata

Services log metadata but forget to include performance metrics:

{
  "message": "user login success",
  "metadata": {
    "db_duration_ms": 14,
    "cache_hit": true
  }
}

But they neglect request_size_bytes, user_agent, or worker_id. Over time, key metrics are missing.

5. Overly Nested Error Details

The details field becomes deeply nested:

"error": {
  "details": {
    "request": {
      "headers": {
        "Authorization": "Bearer abc123",
        "Content-Type": "application/json"
      },
      "body": {
        "email": "[email protected]"
      }
    }
  }
}

This makes querying and alerting complex. Tools must flatten the structure, or queries become brittle.

Best Practices

Tools and Integration

curl -X POST http://logs.example.com/ingest \
  -H "Content-Type: application/json" \
  -d '{
    "timestamp": "2023-08-15T12:34:56.789Z",
    "level": "info",
    "service": "user-service",
    "message": "user created",
    "user_id": 123
  }'
kubectl logs -f deployment/user-service --tail=50
SELECT * FROM logs
WHERE service = 'auth-service'
  AND level = 'error'
  AND timestamp > '2023-08-15'
ORDER BY timestamp DESC
LIMIT 100;
systemctl restart fluentd
systemctl status fluentd --lines=100

Structured logging is not a one-time setup. It is a continuous practice of consistency, correctness, and observability. The goal is to make logs not just a record of what happened, but a tool for understanding, diagnosing, and improving systems.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.