Service Mesh: Istio, Linkerd & Beyond

A service mesh is a dedicated infrastructure layer that manages service-to-service communication in a cloud-native environment. It handles observability, traffic management, security, and policy enforcement across a distributed system of microservices. This becomes essential when services span multiple clusters, regions, or platforms, and when you need consistent behavior—such as retries, circuit breaking, or mutual TLS—across all services without modifying their code. It matters when you have more than a dozen services, each deployed independently, and you need to monitor, route, and secure them uniformly.

Traffic Management

Traffic routing in a service mesh is driven by declarative configuration that governs how requests flow between services. The mesh uses a sidecar proxy (typically Envoy) injected into each service pod, which intercepts all incoming and outgoing traffic.

Virtual Services and Destination Rules

Traffic is controlled via VirtualService and DestinationRule resources in Kubernetes.


apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: productpage
spec:
  hosts:
    - productpage
  http:
    - match:
        - uri:
            exact: /details
      route:
        - destination:
            host: details
            subset: v1
        - weight: 70
      route:
        - destination:
            host: details
            subset: v2
        - weight: 30
    - match:
        - uri:
            prefix: /reviews
      route:
        - destination:
            host: reviews
            subset: v1
          headers:
            request:
              add:
                "x-tracing-enabled": "true"

This configuration routes 70% of /details requests to details:v1 and 30% to details:v2, and forwards all /reviews traffic to reviews:v1, adding a header that enables tracing.

Canary Deployments

Canary deployments are implemented using weighted routing and header-based matching.

# canary-deployment.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: ratings
spec:
  hosts:
    - ratings
  http:
    - match:
        - headers:
            cookie:
              regex: "user=.*"
      route:
        - destination:
            host: ratings
            subset: v2
          weight: 100
    - match:
        - headers:
            cookie:
              regex: "user=(.+)@example.com"
      route:
        - destination:
            host: ratings
            subset: v3
          weight: 100
    - route:
        - destination:
            host: ratings
            subset: v1
          weight: 100

Traffic is routed to ratings:v2 for users with any cookie, and to ratings:v3 for users from example.com. This enables gradual rollouts based on user identity or session data.

Fault Injection

Fault injection is used to test resilience and validate circuit-breaking behavior.

# fault-injection.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: user-service
spec:
  hosts:
    - user-service
  http:
    - match:
        - uri:
            prefix: /profile
      fault:
        delay:
          fixedDelay: 2s
          percentage:
            value: 50
        abort:
          httpStatus: 503
          percentage:
            value: 30

This introduces a 2-second delay on 50% of all /profile requests and aborts 30% of them with a 503 status. The sidecar proxy applies these rules transparently, without requiring changes to the service code.

Observability

Observability is a core function of service meshes, providing distributed tracing, metrics, and logging across services.

Distributed Tracing

Tracing is enabled via propagation of a trace context across service boundaries. The mesh injects traceparent and tracestate headers into HTTP requests.

GET /order HTTP/1.1
Host: order-service
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate: otel.trace_id=0af7651916cd43dd8448eb211c80319c,otep.span_id=b7ad6b7169203331

The traceparent header contains the trace ID and span ID. The tracestate includes vendor-specific metadata. The sidecar ensures these are set on every outgoing request and propagated on every incoming request.

Metrics Collection

Metrics are collected by the sidecar and aggregated by the control plane. Key metrics include:

istio_requests_total – total requests per service, method, and response code
istio_request_duration_milliseconds – latency per service and route
istio_tcp_received_bytes_total – inbound and outbound traffic volume

These are exported via Prometheus, and queries can be written directly:

# Request rate by service, with 95th percentile latency
rate(istio_requests_total{destination_service=~".*", response_code="200"}[5m])
  * on(source_service) group_left
  histogram_quantile(0.95, sum(rate(istio_request_duration_milliseconds_bucket[5m])) by (source_service, destination_service, le))

Logging and Structured Logs

Logs are enriched with service context and trace identifiers.

{
  "timestamp": "2024-05-10T12:34:56.789Z",
  "level": "info",
  "message": "Order processed",
  "traceId": "0af7651916cd43dd8448eb211c80319c",
  "spanId": "b7ad6b7169203331",
  "sourceService": "order-service",
  "destinationService": "payment-service",
  "requestId": "req-abc123"
}

The sidecar writes structured logs to standard output, and the control plane collects and correlates logs across services.

Security

Service mesh security is built around mutual TLS (mTLS), identity-based access control, and policy enforcement.

Mutual TLS

mTLS is enabled per service using PeerAuthentication and DestinationRule.

# peer-authentication.yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: STRICT

This enforces mTLS between all services in the mesh. The control plane issues certificates to sidecars via Istiod, and the sidecar validates peer certificates.

When mTLS is enabled, the sidecar performs certificate validation and sends TLS ClientHello with the service’s identity. The server validates the client certificate and establishes a secure channel.

Service-to-Service Authorization

Access policies are defined using AuthorizationPolicy.

# authorization-policy.yaml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: user-service-access
spec:
  selector:
    matchLabels:
      app: user-service
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/default/sa/user-service-sa"]
      to:
        - operation:
            methods: ["GET", "POST"]
            paths: ["/profile", "/settings"]
      when:
        - key: request.headers[Authorization]
          values: ["Bearer .*"]

This policy allows only requests from the user-service-sa service account to access /profile and /settings, provided the Authorization header contains a Bearer token.

JWT Validation

For external authentication, the mesh validates JWT tokens in the Authorization header.

# jwt-validation.yaml
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: jwt-auth
spec:
  jwt:
    issuer: "https://auth.example.com"
    jwksUri: "https://auth.example.com/.well-known/jwks.json"
    jwtHeaders:
      - "Authorization"
    jwtClaim:
      - "sub"
      - "email"
    issuer:
      - "https://auth.example.com"
    audiences:
      - "api.example.com"

The sidecar extracts the JWT from the Authorization header, validates the signature using the JWKs, and enriches the request context with claims.

Configuration and Deployment

The control plane manages configuration across the mesh. It deploys and scales the sidecar proxies, distributes policies, and monitors the mesh state.

Sidecar Injection

Sidecar proxies are injected into pods using a mutating admission webhook.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews
  labels:
    app: reviews
spec:
  replicas: 3
  selector:
    matchLabels:
      app: reviews
  template:
    metadata:
      labels:
        app: reviews
        version: v1
      annotations:
        "sidecar.istio.io/inject": "true"
        "sidecar.istio.io/rewriteAppHTTPProbes": "true"
    spec:
      containers:
        - name: reviews
          image: reviews:v1
          ports:
            - containerPort: 9080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 9080
          livenessProbe:
            httpGet:
              path: /healthz
              port: 9080

The sidecar.istio.io/inject annotation triggers injection, and rewriteAppHTTPProbes configures the sidecar to use its own health checks.

Control Plane Scaling

The control plane consists of Istiod (or Linkerd control plane), which runs the configuration and policy engines.

# Scale Istiod to handle 1000 services
kubectl scale deployment istiod -n istio-system --replicas=5

Istiod watches Kubernetes CRDs and generates Envoy configuration for each sidecar. It pushes configuration via xDS (Envoy’s discovery service) protocols: LDS (Listener Discovery), RDS (Route Discovery), CDS (Cluster Discovery), and SDS (Secret Discovery).

Configuration Drift and Failure Modes

Configuration drift occurs when the control plane diverges from the actual state in the mesh. Common causes:

A VirtualService is applied but not synced to the sidecar proxies.
A DestinationRule is applied but not reflected in Envoy’s route configuration.
A PeerAuthentication policy is applied but not pushed to the sidecar via xDS.

Failure modes include:

mTLS handshake failure: sidecar fails to validate peer certificate; logs show TLS alert: handshake failure and certificate verify failed.
xDS configuration not received: Envoy logs no xDS response received and falls back to static config.
Trace context missing: spans are created but not linked; traceparent header is missing or malformed.
JWT validation fails: invalid JWT signature, audience mismatch, or issuer mismatch.

Advanced Patterns

Circuit Breaking

Circuit breaking prevents overload cascades. It is configured via DestinationRule.

# circuit-breaking.yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: user-service
spec:
  host: user-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
        idleTimeout: 300s
      http:
        http1MaxPendingRequests: 100
        maxRequestsPerConnection: 10
    outlierDetection:
      consecutive5xxErrors: 5
      baseEjectionTime: 30s
      interval: 10s
      maxEjectionPercent: 50

When five consecutive 5xx errors occur, the service is ejected from the pool. The circuit breaker opens and routes traffic to healthy instances, and the control plane gradually re-introduces the service.

Global Traffic Management

For multi-cluster deployments, the mesh uses a global Gateway and VirtualService to route traffic across clusters.

# global-gateway.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: global-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "api.example.com"

The control plane routes traffic to the appropriate cluster based on source, headers, or load.

Troubleshooting

Inspecting Sidecar Configuration

Use istioctl proxy-config to inspect sidecar state.

# View Envoy configuration for a sidecar
istioctl proxy-config cluster reviews-v1-789df456c7 -n default
istioctl proxy-config route reviews-v1-789df456c7 -n default
istioctl proxy-config listener reviews-v1-789df456c7 -n default

This shows the full xDS configuration and helps debug routing, mTLS, and load balancing.

Debugging mTLS

Use curl to check mTLS handshake:

curl -H "Authorization: Bearer abc123" https://user-service -v -H "Host: user-service" --cert-type PEM --cert user.crt --key user.key

The sidecar logs show TLS handshake completed, peer verified, and certificate verified.

Validating JWTs

Check JWTs with jwt.io or validate via openssl:

# Decode JWT
echo "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJmB1t3X20Xo3331763823" | \
  jq -R 'split(".") | .[1] | @base64d | fromjson'

This reveals the JWT payload and helps verify claims and signatures.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.