Distributed Tracing: Following Requests Across Service Boundaries

Implement distributed tracing to debug latency, identify bottlenecks, and understand request flow across microservices. Covers OpenTelemetry, trace propagation, span design, sampling strategies, and integrating traces with logs and metrics for full observability.

Requests in a modern backend system often span multiple services, each deployed independently and communicating via HTTP, message queues, or gRPC. When a user performs an action — say, placing an order — the request flows from a frontend service through an API gateway, into a product catalog, an inventory service, a payment processor, and finally a notification system. Each service logs its own events, but without a unifying thread, diagnosing latency, failures, or inconsistencies becomes a guessing game. Distributed tracing provides a single, end-to-end timeline of a request as it crosses service boundaries, stitching together individual logs, metrics, and spans of execution.

Correlating Spans Across Services

To follow a request through a distributed system, each operation — a function call, an HTTP request, a database query — is recorded as a *span*. A span includes a name, start and end timestamps, and optional metadata such as tags, logs, and baggage. The key to distributed tracing is the trace ID, a globally unique identifier that ties all spans related to a single request.

Every service in the chain must propagate the trace ID across boundaries. When a service receives a request, it creates a new span and adds the trace ID to the request headers. If the service makes a downstream call, it embeds the trace ID and a *span ID* (a unique ID for the current span) in the outgoing request.

The standard header format is:

X-Trace-ID: 7f2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e
X-Span-ID: 1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d

Most frameworks automatically extract the trace ID from incoming requests and use it to create a new span. The span ID is generated locally and attached to the outgoing request.

If a service receives a request with X-Trace-ID, it must:

  1. Extract the trace ID.
  2. Create a new span with that trace ID and a unique span ID.
  3. Record the span’s start time.
  4. Propagate the trace and span IDs to the next service.

Failure here is silent but common: a service may log a span but forget to propagate the trace ID, resulting in disjointed spans that cannot be stitched together. A span with no parent span ID is a *root span*, typically created by the entry point service (e.g., the API gateway).

Propagating Context Across Communication Protocols

HTTP is the most common transport, and the propagation of trace context via headers is well established. However, each protocol has its own way of carrying the trace context.

In gRPC, the trace context is carried in the metadata map of the RPC call. The following keys are used:

A client service sending a gRPC request must set these metadata entries before calling the remote method.

In message queues like Apache Kafka or Amazon SQS, the trace context is embedded in the message payload or in the message headers.

For Kafka, the trace context is typically added to the record headers using the following standard keys:

{
  "trace-id": "7f2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e",
  "span-id": "1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d",
  "parent-span-id": "5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d"
}

When a consumer processes a Kafka message, it extracts the trace context and creates a new span. If the consumer makes an HTTP call, it injects the trace context into the outgoing headers.

For event-driven systems using a message broker like RabbitMQ, the context is passed in the message properties. The headers field of the AMQP message is used to carry the trace context as a map of strings.

Common failures include:

Instrumenting Services with Spans

Each service must create spans for key operations. The most common pattern is to use middleware or a framework-provided tracing layer.

In a typical HTTP server, the tracing middleware:

  1. Extracts the trace ID from incoming headers.
  2. Creates a new span for the HTTP request.
  3. Records the start time.
  4. Sets the span as current.
  5. Calls the next handler.
  6. On completion, records the span’s end time and logs any relevant metadata.

In Node.js (Express), a typical setup uses the opentelemetry-js library:

const { NodeTracerProvider } = require('@opentelemetry/sdk-node');
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
const { ConsoleSpanExporter } = require('@opentelemetry/sdk-trace-node');

const provider = new NodeTracerProvider();
provider.addSpanProcessor(new ConsoleSpanExporter());
provider.register();

const tracer = provider.getTracer('my-app');

app.use((req, res, next) => {
  const span = tracer.startSpan('HTTP Request', {
    attributes: {
      'http.method': req.method,
      'http.url': req.url,
      'http.user_agent': req.get('User-Agent'),
    },
  });

  // Set span as current
  const ctx = trace.setSpan(context.active(), span);
  const scope = tracer.with(ctx, () => {
    // Forward the span to the next middleware
    res.on('finish', () => {
      span.end();
    });
    next();
  });

  // Attach span to res.locals
  res.locals.span = span;
});

In Go (Gin), the pattern is similar:

import (
  "context"
  "github.com/gin-gonic/gin"
  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/attribute"
  "go.opentelemetry.io/otel/trace"
)

func TracingMiddleware() gin.HandlerFunc {
  return func(c *gin.Context) {
    ctx := c.Request.Context()
    tracer := otel.Tracer("my-app")

    spanName := c.Request.Method + " " + c.Request.URL.Path
    span, ctx := tracer.Start(ctx, spanName, trace.WithAttributes(
      attribute.String("http.method", c.Request.Method),
      attribute.String("http.url", c.Request.URL.String()),
      attribute.String("http.user_agent", c.GetHeader("User-Agent")),
    ))

    c.Request = c.Request.WithContext(ctx)
    defer span.End()

    c.Next()

    // Record any error from the handler
    if err := c.Errors.Last(); err != nil {
      span.SetAttributes(attribute.String("error", err.Error()))
    }
  }
}

The critical detail is that the span must be *active* during the execution of the operation. If a service makes a database call within the span, the database instrumentation must capture the current span and create a child span.

Observing Traces in Practice

Once spans are emitted, they must be collected and stored. The common pattern is to send spans to a central tracing backend via an HTTP API.

A typical span payload has the following shape:

{
  "traceId": "7f2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e",
  "spanId": "1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d",
  "parentId": "5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d",
  "name": "HTTP Request",
  "startTime": "2023-10-05T12:00:00.000Z",
  "endTime": "2023-10-05T12:00:00.150Z",
  "attributes": {
    "http.method": "POST",
    "http.url": "/orders",
    "http.user_agent": "Mozilla/5.0"
  },
  "events": [
    {
      "name": "db.query",
      "timestamp": "2023-10-05T12:00:00.040Z",
      "attributes": {
        "sql": "SELECT * FROM products WHERE id = ?",
        "rows": 1
      }
    }
  ],
  "status": {
    "code": 200,
    "message": "OK"
  }
}

The service sends this payload via POST /api/traces to a tracing backend. The backend stores spans in a database and exposes a query API.

A typical query to retrieve a trace by ID:

GET /api/traces/7f2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e HTTP/1.1
Host: tracing.example.com
Accept: application/json

The response returns all spans in the trace, ordered by start time, with parent-child relationships.

Common failure modes:

Debugging with Traces

When a request fails, engineers follow the trace to identify the root cause. The first step is to identify the *longest span* — the one that took the most time.

A span with a status.code of 500 or a status.message containing error is a candidate. But the real insight comes from events.

Events are timestamps with additional context. For example, a database query span may include events for:

The timeline of a trace reveals:

A common mistake is assuming that a span’s duration equals the time spent in that service. In reality, the span includes the time spent in downstream calls, unless explicitly recorded.

Best Practices and Sharp Edges

The most silent failure: a service creates a span but forgets to end it. The span remains open, and its end time is set only when the process shuts down. This leads to traces that appear incomplete — spans with endTime set to null or now, and no clear timeline.

Another edge: multiple root spans in a single trace. A service may receive a request and create a trace, but then make a downstream call that itself creates a new trace. Without proper trace ID propagation, two separate traces are created instead of one.

Finally, baggage — key-value pairs that propagate across service boundaries — is underused. A request may carry baggage like user.id=123, session.token=abc123, correlation-id=xyz. These can be used to enrich spans and correlate traces across systems.

Summary

Distributed tracing is the practice of following a request across multiple services, using trace IDs to stitch together spans from different systems. The core mechanism relies on consistent propagation of trace and span IDs across HTTP, gRPC, message queues, and other protocols. Each service must instrument key operations, creating spans with meaningful attributes and events. The resulting trace provides a timeline of execution, revealing latency, failures, and interactions. The most common pitfalls — missing trace IDs, unended spans, incorrect propagation — are silent but devastating. Mastering distributed tracing requires attention to detail in how context is passed, how spans are created, and how events are recorded.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.