Observability Stack: Logs, Metrics, Traces Unified

This page describes how to build and operate a unified observability stack using standard tools and protocols. It focuses on integrating logs, metrics, and traces into a cohesive system that can be deployed without vendor lock-in or proprietary tooling.

When to Use This Approach

Use this approach when building systems that require consistent visibility across multiple services, where you need to correlate data between logging, metrics, and tracing systems. The unified stack works best when you have a small to medium-sized deployment with shared infrastructure, and when the cost of vendor-specific tooling outweighs its benefits.

Setting Up Centralized Logging

Configure Log Aggregation

Log aggregation should use standard structured formats such as JSON. Example log line:

{"timestamp":"2023-06-15T14:30:45.123Z","level":"INFO","service":"user-service","message":"User login successful","user_id":"12345"}

Use a log forwarder like Fluentd or Vector to collect logs from all services:

# Example vector configuration for log collection
[sources.main]
type = "file"
include = ["/var/log/app/*.log"]

Log Retention and Rotation

Configure log rotation with logrotate:

/var/log/app/application.log {
    rotate 7
    daily
    compress
    delaycompress
    missingok
    notifempty
    create 0644 root root
}

Collecting and Storing Metrics

Expose Application Metrics

Use standard HTTP endpoints for metrics exposure. For example, a Go application using Prometheus client:

import "github.com/prometheus/client_golang/prometheus"

var (
    httpRequestDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name: "http_request_duration_seconds",
            Help: "HTTP request duration in seconds",
        },
        []string{"method", "endpoint"},
    )
)

func init() {
    prometheus.MustRegister(httpRequestDuration)
}

Configure Metric Collection

Metrics should be scraped at regular intervals. Example configuration for Prometheus:

scrape_configs:
  - job_name: 'application'
    static_configs:
      - targets: ['localhost:8080']

Implementing Distributed Tracing

Instrument Code for Tracing

Use OpenTelemetry SDK to add tracing to applications. Example in Python:

from opentelemetry import trace
from opentelemetry.trace import SpanKind

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("process_order", kind=SpanKind.SERVER):
    # Business logic here
    pass

Trace Export Configuration

Traces should export to a collector that supports OpenTelemetry Protocol (OTLP):

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging]

Unified Query Interface

Creating Correlation Points

Use a consistent identifier across logs, metrics, and traces. The trace ID is the most common:

{"trace_id":"a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6","span_id":"1234567890abcdef"}

Querying Across Systems

All three data types should be queryable through a single interface. For example, using the jq tool to extract trace IDs from logs:

grep "trace_id" /var/log/app/application.log | jq -r '.trace_id' | sort | uniq

Data Storage and Retention

Time-Series Database Setup

Use Prometheus for metrics storage with retention configuration:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alert.rules"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

Log Storage Configuration

For logs, use a standard file-based system with compression:

# Example log directory structure
/logs/
  /application/
    application.log
    application.log.1.gz
    application.log.2.gz

Alerting and Notification

Defining Alerts

Alerts should be defined using PromQL expressions:

groups:
  - name: application-alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.01
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "High error rate detected"

Notification Integration

Configure alertmanager to send notifications via webhook:

route:
  receiver: 'webhook'

receivers:
  - name: 'webhook'
    webhook_configs:
      - url: 'http://alertmanager.example.com/webhook'

Operational Considerations

Performance Impact

Logs, metrics, and traces can significantly impact system performance. Monitor CPU and memory usage of collection agents. A misconfigured log forwarder can cause 10% overhead on application performance.

Resource Constraints

Limit the amount of data collected:

# Example fluentd filter to limit log volume
<filter application.log>
  @type grep
  <regexp>
    key message
    pattern ^INFO
  </regexp>
</filter>

Data Loss Prevention

Configure buffering for transient failures:

# Vector configuration with buffer
[sources.main]
type = "file"
include = ["/var/log/app/*.log"]
buffer = { type = "disk", max_size = 1073741824 }

Common Failure Modes

Log Forwarder Disconnection

If a log forwarder loses connection, logs can be lost silently. Monitor the forwarder's connection status and set up alerts for disconnections.

Metric Scraping Failures

Prometheus scraping failures can be silent. Configure health checks to detect missing metrics:

curl -f http://localhost:8080/metrics || echo "Metrics endpoint down"

Trace Ingestion Issues

Tracing systems can drop spans during high load. Monitor queue depths and set up alerts for backpressure.

Correlation Breakdown

Trace IDs can be missing from logs, causing correlation failures. Implement strict logging policies where all log entries must include trace context.

Integration Testing

Testing End-to-End Flow

Create a test that verifies the complete flow:

# Test that logs contain trace IDs
curl -s http://localhost:8080/endpoint | jq -r '.trace_id' > /tmp/trace_id

# Verify logs contain that trace ID
grep "$(cat /tmp/trace_id)" /var/log/app/application.log

Testing Alerting Rules

Verify alert rules with synthetic data:

# Simulate high error rate
curl -X POST http://localhost:8080/metrics/increment?error=1

Configuration Management

Version Control for Configurations

Store all configuration files in version control:

# Example directory structure
/configs/
  /prometheus/
    prometheus.yml
  /vector/
    vector.toml
  /alertmanager/
    alertmanager.yml

Environment-Specific Settings

Use environment variables to adjust configurations:

export PROMETHEUS_RETENTION=30d
export LOG_LEVEL=INFO

Monitoring the Observability Stack

Health Checks for Components

Monitor each component with its own health endpoint:

# Check Prometheus
curl http://localhost:9090/-/healthy

# Check Vector
curl http://localhost:9001/health

Metrics About Metrics

Collect metrics about your observability stack itself:

var (
    collectorUp = prometheus.NewGaugeVec(
        prometheus.GaugeOpts{
            Name: "collector_up",
            Help: "Whether the collector is up",
        },
        []string{"component"},
    )
)

Migration Considerations

Moving from Legacy Systems

When migrating from a legacy logging system, ensure all historical data can be indexed:

# Example script to reindex old logs
for file in /old/logs/*.log; do
  zcat "$file" | jq '.trace_id |= . // ""' > "${file}.new"
done

Backward Compatibility

Maintain compatibility with existing alerting systems by supporting multiple output formats.

Security and Access Control

Log File Permissions

Ensure logs have appropriate permissions:

chmod 640 /var/log/app/application.log
chown root:app /var/log/app/application.log

Metric Endpoint Protection

Protect metrics endpoints from unauthorized access:

# Example nginx configuration
location /metrics {
    allow 10.0.0.0/8;
    deny all;
}

Maintenance Tasks

Regular Cleanup

Schedule regular cleanup of old logs and metrics:

# Remove logs older than 30 days
find /var/log/app -name "*.log" -mtime +30 -delete

Configuration Updates

Update configurations in a controlled manner with rollbacks available.

Troubleshooting

Debugging Correlation Issues

Use trace ID to cross-reference between systems:

# Find all entries for a specific trace ID
grep "a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6" /var/log/app/*.log

Performance Degradation

Monitor resource usage of collection agents and adjust accordingly.

The unified observability stack requires careful attention to configuration, consistent data formats, and robust error handling. The key is to ensure that logs, metrics, and traces all work together without introducing complexity or silent failures.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.