Distributed Tracing

Distributed tracing tracks requests as they flow through systems composed of multiple services, allowing engineers to understand request behavior, find bottlenecks, and debug performance issues across service boundaries. It matters when requests span more than one process, especially when latency is unexpected or when debugging failures that cross service boundaries.

Understanding Trace Context Propagation

Traces begin with a root span that carries a trace ID. When services receive a request, they must extract the trace context from incoming headers and continue the trace. The standard header format is:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-00f067aa0ba902b7-01
tracestate: rojo=00-0af7651916cd43dd8448eb211c80319c-00f067aa0ba902b7-01

The traceparent header contains version, trace ID, span ID, and trace flags. The tracestate header carries vendor-specific baggage. When propagating traces, services must preserve both headers exactly as received.

Creating a Trace Span

To create a new span in code:

// Example in Go using OpenTelemetry SDK
span := tracer.Start(ctx, "operation-name")
defer span.End()

// Add attributes to the span
span.SetAttributes(
    attribute.String("http.method", "GET"),
    attribute.String("http.url", req.URL.String()),
)

The span ID must be unique within a trace. When creating child spans, the new span's ID becomes the parent's span ID in the next span's context.

Configuring Trace Exporters

Traces are exported to backend systems using exporters. The most common exporter is the OTLP (OpenTelemetry Protocol) exporter, which sends data over gRPC or HTTP.

For HTTP-based export:

# Configuration for OpenTelemetry collector
exporters:
  otlp:
    endpoint: "http://otel-collector:4318"
    tls:
      insecure: true

For gRPC-based export:

exporters:
  otlp:
    endpoint: "otel-collector:4317"
    tls:
      insecure: false

The OTLP exporter sends spans as protobuf-encoded data. The protocol uses a fixed schema that must be maintained across versions.

Using Trace Context in HTTP Requests

When making outbound requests, services must inject the trace context into outgoing headers:

// Example in Go
func makeRequest(ctx context.Context, client *http.Client, url string) error {
    req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
    
    // Inject trace context
    carrier := propagation.HeaderCarrier(req.Header)
    propagator := propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    )
    propagator.Inject(ctx, carrier)
    
    _, err := client.Do(req)
    return err
}

When receiving requests, extract the trace context:

func handleRequest(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    
    // Extract trace context
    propagator := propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    )
    
    ctx = propagator.Extract(ctx, propagation.HeaderCarrier(r.Header))
    
    // Start a new span with extracted context
    span := tracer.Start(ctx, "handler")
    defer span.End()
}

Setting Up Trace Sampling

Trace sampling controls how many traces are recorded. The default is often 100% sampling, but this can be expensive.

Sample configuration:

# OpenTelemetry collector configuration
processors:
  probabilistic:
    sampling_percentage: 10

Sampling decisions happen at the root span level. If a trace is sampled, all spans in that trace are exported. If not sampled, no spans from that trace are exported.

Trace Data Structure

Each span contains these key fields:

• trace_id: 16-byte identifier for the entire trace
• span_id: 8-byte identifier for this specific span
• parent_span_id: 8-byte identifier of the parent span (0 for root spans)
• name: Human-readable operation name
• kind: Type of span (client, server, producer, consumer)
• start_time: When the span started
• end_time: When the span ended
• attributes: Key-value pairs describing the span
• events: Timestamped events during the span's lifetime

Handling Trace Context Loss

Trace context can be lost when services don't properly propagate headers. This commonly happens when:

1. An HTTP client doesn't send trace headers
2. A service receives requests without trace context
3. Headers are modified or dropped by intermediaries
4. Services use different propagation formats

When trace context is missing, a new trace is started. The system should log these occurrences:

WARN: Trace context missing in request to /api/users

Correlating Traces Across Services

To correlate traces across services, ensure that trace IDs are preserved and propagated through all services. A trace ID must be 16 bytes (32 hex characters) for compatibility with W3C Trace Context.

When a service receives a request with an invalid trace ID, it should log the error and continue with a new trace:

ERROR: Invalid trace ID format in header "traceparent"

Debugging Trace Issues

To debug tracing issues, check these common failure modes:

1. Missing spans: Ensure all services are configured to export traces
2. Incomplete traces: Check that trace context is propagated through all service boundaries
3. Inconsistent timestamps: Verify that clocks across services are synchronized
4. Trace loss: Confirm sampling rates and exporter configurations

Use the following query in your tracing backend to find incomplete traces:

SELECT trace_id, count(*) as span_count 
FROM spans 
WHERE end_time IS NULL 
GROUP BY trace_id 
HAVING count(*) < 5

Trace Retention and Storage

Traces are stored in backends with different retention policies. Common retention settings include:

• 7 days for production traces
• 30 days for debugging traces
• 90 days for compliance traces

Storage systems may have limits on trace duration, span count, or attribute size. Traces that exceed these limits are truncated.

Trace Exporting Failures

When exporting traces fails, the system should log the error:

ERROR: Failed to export trace to collector: connection refused

The exporter should retry failed exports with exponential backoff. If retries fail after a configured number of attempts, the trace data should be dropped.

Instrumentation Best Practices

1. Name spans meaningfully: Use operation names that describe what happened
2. Add relevant attributes: Include request parameters, response codes, and error details
3. Set appropriate span kinds: Distinguish between client, server, producer, and consumer spans
4. Avoid nested spans: Keep spans shallow to avoid performance overhead
5. End spans explicitly: Always call span.End() to ensure proper cleanup

Trace Context in Queue-Based Systems

When using message queues, trace context must be preserved across the message lifecycle:

// Example: Sending a message with trace context
func sendMessage(ctx context.Context, queue string, message []byte) error {
    carrier := propagation.HeaderCarrier{}
    propagator := propagation.TraceContext{}
    propagator.Inject(ctx, carrier)
    
    // Add trace headers to message metadata
    msg := &amqp.Publishing{
        Body:       message,
        Headers:    amqp.Table{},
    }
    
    for key, values := range carrier {
        for _, value := range values {
            msg.Headers[key] = value
        }
    }
    
    return channel.Publish(queue, "", false, false, msg)
}

Handling Trace Context in Asynchronous Operations

Asynchronous operations require careful handling of trace context:

// Example: Continuing trace in background goroutine
func processAsync(ctx context.Context) {
    // Extract context from original request
    propagator := propagation.TraceContext{}
    ctx = propagator.Extract(ctx, propagation.HeaderCarrier(r.Header))
    
    go func() {
        // Create new span for async work
        span := tracer.Start(ctx, "async-operation")
        defer span.End()
        
        // Do async work...
    }()
}

Trace Metadata and Attributes

Common attributes to add to spans include:

• http.method: HTTP method (GET, POST, etc.)
• http.url: Full URL of the request
• http.status_code: HTTP response code
• error.type: Exception type if error occurred
• error.message: Error message
• db.statement: SQL statement executed
• service.name: Name of the service processing the request

Trace Compression and Limits

Trace systems often compress spans to reduce network overhead. The compression algorithm must be consistent across all services. Most tracing systems limit:

• Maximum number of spans per trace (typically 1000)
• Maximum attribute value size (typically 4KB)
• Maximum span duration (typically 1 hour)

Trace Filtering and Aggregation

Some backends support filtering traces by attributes or sampling. The filter expression is typically a string that follows the backend's query language.

Example filter:

http.status_code = 500 AND service.name = "payment-service"

Troubleshooting Common Issues

Inconsistent Trace IDs

When spans from the same request show different trace IDs, check for:

1. Missing propagation in HTTP clients
2. Incorrect header handling
3. Service restarts that lose context
4. Middleware that modifies headers

Span Duration Mismatches

If span durations seem wrong, verify:

1. Clock synchronization across services
2. Proper span end calls
3. Time zone settings in trace storage

Exporter Configuration Errors

When traces don't appear in the backend:

1. Check connection to collector endpoint
2. Verify TLS configuration
3. Confirm exporter type matches backend expectations
4. Review exporter logs for errors

Trace Data Validation

Valid trace data should follow these rules:

• All spans must have valid trace_id and span_id
• Parent_span_id must reference an existing span in the same trace
• Start time must be before end time
• Required attributes must be present where expected
• Span kind must be one of: client, server, producer, consumer, internal

Trace Exporter Health Monitoring

Monitor exporter health by:

1. Checking export success rate
2. Monitoring connection status to backend
3. Tracking dropped trace count
4. Verifying data format compliance

Trace Context Header Variations

Different systems may use different header names for trace context:

• Standard: traceparent, tracestate
• Cloud-specific: x-cloud-trace-context
• Legacy: uber-trace-id

Ensure consistent header handling across services to prevent trace fragmentation.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.