Logging Best Practices

Production engineering guide for logging best practices covering patterns, implementation strategies, and operational best practices.

Logging Best Practices is a guide to writing, managing, and consuming logs in production software systems. It applies to any service, batch job, or infrastructure component where visibility into execution is essential. Use this guide when debugging a failure that spans services, when onboarding developers to a new system, or when setting up a new environment where logs are the primary source of truth.

Centralizing Logs

Logs must be centralized to be useful. A service deployed across multiple machines, containers, or cloud instances cannot be debugged by inspecting individual file logs. The goal is to aggregate all logs into a single searchable stream where every event is indexed and time-ordered.

Use a logging pipeline that collects logs from each host or process and sends them to a central store. The pipeline should be resilient to network outages and should buffer logs locally when the central system is unreachable.

The minimal pipeline consists of a log forwarder daemon running on each host. It reads from standard output, file logs, or structured log files, and forwards them via a lightweight protocol to a central collector.

For example, a service written in Go outputs structured JSON logs to stdout. A systemd service file ensures it starts on boot and restarts on failure:

[Unit]
Description=My Service
After=network.target

[Service]
ExecStart=/usr/local/bin/my-service
StandardOutput=journal
StandardError=journal
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

The forwarder, such as fluent-bit, collects logs from journal and sends them to a remote HTTP endpoint:


[INPUT]
    Name              systemd
    Tag               my-service
    Systemd_Filter    _SYSTEMD_UNIT=my-service.service

[OUTPUT]
    Name              http
    Match             my-service
    Host              logs.example.com
    Port              8080
    Format            json
    Json.Keys         time, message, level, component, trace_id, user_id
    Retry_Limit       10
    Buffer_Size       1MB

This setup ensures that every log line from every instance of the service appears in the central system within seconds. If the central logs server is down, fluent-bit buffers logs locally until it reconnects.

Failure Mode: Log Loss During Outage

When the central logging server is down, logs accumulate in fluent-bit’s local buffer. If the buffer fills, logs are dropped. To prevent this, configure fluent-bit to use a persistent file buffer:

[OUTPUT]
    Name              http
    Match             my-service
    Host              logs.example.com
    Port              8080
    Format            json
    Buffer_Size       1MB
    Buffer_Max_Size   10MB
    Buffer_Path       /var/lib/fluent-bit/buffer
    Buffer_Checksum   on

With Buffer_Checksum, fluent-bit ensures that each batch of logs is delivered exactly once. If a batch fails, it is retried until success. The Buffer_Max_Size setting prevents data loss during extended outages.

Structured Logging

Logs should not be free-form text. Every log entry should be a structured record, not a string. Use JSON or another format that supports key-value pairs, arrays, and nested objects.

A well-structured log includes at minimum:

Example log entry:

{
  "timestamp": "2024-03-15T12:34:56.789Z",
  "level": "error",
  "message": "Failed to process user payment",
  "trace_id": "a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8",
  "user_id": "u987654321",
  "payment_id": "p432109876",
  "error": {
    "code": "PAYMENT_PROCESSING_FAILED",
    "details": "Transaction timed out after 30s",
    "stack_trace": [
      "at PaymentProcessor.Process(PaymentRequest request)",
      "at PaymentService.HandlePaymentAsync(Guid paymentId)",
      "at ApiGateway.HandleRequest(HttpRequest request)"
    ]
  },
  "duration_ms": 31245,
  "retries": 3
}

This structure enables powerful queries and alerts.

Failure Mode: Mixing String and Structured Logs

When logs are mixed—some are plain strings, others are JSON—searching and alerting become brittle. A common mistake is emitting printf-style logs from a service:

log.Printf("User %s failed to authenticate with token %s", userID, token)

This produces a string, but the key-value context is lost. To fix this, use a logging library that supports structured logging. In Go, use logr or zap:

import "go.uber.org/zap"

func ProcessPayment(ctx context.Context, payment Payment) error {
    logger := zap.L()
    start := time.Now()

    if err := db.Save(payment); err != nil {
        logger.Error("Failed to save payment",
            zap.String("payment_id", payment.ID),
            zap.String("user_id", payment.UserID),
            zap.Error(err),
            zap.Duration("took", time.Since(start)),
        )
        return err
    }

    logger.Info("Payment processed",
        zap.String("payment_id", payment.ID),
        zap.String("user_id", payment.UserID),
        zap.Float64("amount", payment.Amount),
        zap.Duration("took", time.Since(start)),
    )

    return nil
}

The zap logger emits structured JSON logs. The logger.Error call generates a log entry with message, level, timestamp, and the provided fields.

Standardizing Log Levels

Use consistent log levels across all services. Define a canonical set of levels and enforce them.

Each level should be used in a predictable way:

Failure Mode: Misusing Log Levels

A common failure is using info for everything and reserving error for only the most severe failures. This leads to a log stream where info logs are too verbose, and error logs are sparse and hard to correlate.

Use error for any operation that fails and returns a Result or Error. Use warn for operations that succeed but take longer than expected.

For example, a service that fetches data from an external API:

func FetchUserData(userID string) (User, error) {
    start := time.Now()

    logger.Info("Fetching user data",
        zap.String("user_id", userID),
    )

    resp, err := client.Get(fmt.Sprintf("/users/%s", userID))
    if err != nil {
        logger.Error("Failed to fetch user data",
            zap.String("user_id", userID),
            zap.Error(err),
            zap.Duration("took", time.Since(start)),
        )
        return User{}, err
    }

    if resp.StatusCode != http.StatusOK {
        logger.Warn("User data fetch returned non-200 status",
            zap.String("user_id", userID),
            zap.Int("status", resp.StatusCode),
            zap.Duration("took", time.Since(start)),
        )
    }

    // ... parse response ...

    logger.Info("User data fetched successfully",
        zap.String("user_id", userID),
        zap.Int("data_size_bytes", len(data)),
        zap.Duration("took", time.Since(start)),
    )

    return user, nil
}

This use of levels enables observability: info logs show progress; warn logs highlight performance issues; error logs are the primary entry point for failure investigation.

Correlating Logs Across Services

Logs from different services must be linked. Use trace_id to correlate events across services, queues, databases, and external APIs.

A trace_id is a unique identifier that spans a request or operation from start to finish. It is passed through each service in the call chain.

In a typical request flow:

  1. API Gateway receives a request and generates a trace_id.
  2. The gateway logs the incoming request with the trace_id.
  3. The gateway forwards the request to a service, including the trace_id in headers.
  4. Each service logs its own events with the same trace_id.

Use a standard header like X-Trace-ID or traceparent (from OpenTelemetry).

Example HTTP request:

POST /api/users HTTP/1.1
Host: api.example.com
X-Trace-ID: a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8
Content-Type: application/json

{
  "name": "Alice",
  "email": "[email protected]"
}

The service reads the header and uses it in all logs:

func HandleUserRequest(w http.ResponseWriter, r *http.Request) {
    traceID := r.Header.Get("X-Trace-ID")
    logger.Info("Received user creation request",
        zap.String("trace_id", traceID),
        zap.String("method", r.Method),
        zap.String("path", r.URL.Path),
    )

    // ... process request ...

    logger.Info("User created successfully",
        zap.String("trace_id", traceID),
        zap.String("user_id", userID),
    )
}

Failure Mode: Trace Ids Lost in Middleware

A common failure is missing trace_id propagation in middleware or load balancers. The X-Trace-ID header is set at the gateway, but the service does not read it, or the service sets a new trace_id instead of using the incoming one.

To ensure trace propagation, use a middleware that extracts the trace_id from the request and sets it in the logger context:

func TraceMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        traceID := r.Header.Get("X-Trace-ID")
        if traceID == "" {
            traceID = uuid.New().String()
        }

        ctx := context.WithValue(r.Context(), "trace_id", traceID)
        r = r.WithContext(ctx)

        // Set trace_id in logger
        logger := zap.L().With(zap.String("trace_id", traceID))

        // Wrap the response writer to capture trace_id in final logs
        rw := &responseWriter{ResponseWriter: w, traceID: traceID}
        next.ServeHTTP(rw, r)
    })
}

type responseWriter struct {
    http.ResponseWriter
    traceID string
}

func (rw *responseWriter) WriteHeader(code int) {
    rw.ResponseWriter.WriteHeader(code)
    logger.Info("Request completed",
        zap.String("trace_id", rw.traceID),
        zap.Int("status", code),
    )
}

This ensures that every log entry in the request lifecycle carries the same trace_id, enabling full correlation.

Retaining and Archiving Logs

Logs should be retained for a defined period. Use a tiered storage approach: hot (frequent access), warm (less frequent), and cold (long-term archive).

Hot logs: last 7 days, stored in a fast, searchable system like Elasticsearch or OpenSearch.

Warm logs: 30 days, stored in a cost-effective storage like S3 or Google Cloud Storage.

Cold logs: 1 year+, stored in low-cost, archival storage like Glacier or Cold Storage.

Use a log retention policy that moves logs from hot to warm to cold based on age.

For example, with fluent-bit and OpenSearch:

[OUTPUT]
    Name              http
    Match             *
    Host              opensearch.example.com
    Port              9200
    Path              /_bulk
    Format            json
    Retry_Limit       10
    Buffer_Size       1MB
    Buffer_Max_Size   10MB
    Buffer_Path       /var/lib/fluent-bit/buffer
    Buffer_Checksum   on
    Log_Format        json
    HTTP_User         logs
    HTTP_Password     s3cr3t
    HTTP_Header         X-Index-Name: logs-${year}-${month}-${day}

The HTTP_Header creates an index per day. After 7 days, a script runs to roll over the hot index and move it to warm storage.

Failure Mode: Indexing Too Many Logs

When logs are indexed but not partitioned, queries become slow. A single large index with 100 million logs takes minutes to scan. To fix this, partition logs by date and service.

Use a date-based index pattern: logs-2024-03-15, logs-2024-03-16, etc. Each index contains logs from one day. Querying logs for a specific day is fast.

To move logs to warm storage, use a script that:

  1. Identifies indices older than 7 days.
  2. Copies them to S3.
  3. Updates a metadata table to track which indices are in warm storage.
  4. Deletes old indices from the hot system.

This approach reduces query time and storage cost.

Monitoring and Alerting on Logs

Use logs not just for debugging but for proactive monitoring.

Set up alerts based on log patterns, levels, and metrics extracted from logs.

For example, alert when:

Use a log-based alerting system that scans logs for patterns.

Example alert rule in OpenSearch:

{
  "name": "High error rate in user service",
  "query": {
    "bool": {
      "must": [
        { "term": { "service": "user-service" } },
        { "term": { "level": "error" } },
        { "range": { "timestamp": { "gte": "now-10m" } } }
      ]
    }
  },
  "threshold": {
    "count": 10
  },
  "action": {
    "email": "[email protected]"
  }
}

This rule triggers an alert whenever ten or more error logs appear in the user-service over the past ten minutes.

Use grep, awk, and jq to build custom monitoring scripts.

#!/bin/bash
# alert-logs.sh
LOGS_DIR="/var/log/production"
DATE=$(date +%Y-%m-%d)
INDEX_NAME="logs-$DATE"

# Check for high error rate in last hour
errors=$(grep -E 'level.*error' "$LOGS_DIR/$INDEX_NAME.json" | \
        jq -c 'select(.timestamp > (now - 3600))' | \
        wc -l)

if [ "$errors" -gt 50 ]; then
    echo "High error rate: $errors errors in last hour" | \
        mail -s "Alert: High Errors in $INDEX_NAME" [email protected]
fi

This script runs every 10 minutes via cron and sends alerts when the error rate exceeds threshold.

Summary

Logging is not just writing to a file. It is a system of aggregation, structure, correlation, retention, and monitoring. Use structured logs with standard levels. Propagate trace_id across services. Centralize logs with a resilient pipeline. Retain logs across storage tiers. Monitor and alert on log patterns. When logs are done right, they become the primary diagnostic tool in production.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.