Cloud Service Mesh

A cloud service mesh is a dedicated infrastructure layer that handles inter-service communication in a distributed application, removing that logic from the application code. It matters when you need fine-grained traffic management, zero-downtime deployments, service-to-service mTLS, or aggregated observability across dozens of microservices without modifying each one. The mesh interposes a lightweight proxy alongside every service instance, mediates all network calls, and can rewrite, route, or reject traffic based on policies you define centrally. Because the proxies operate at the edge of each service’s network namespace, the application itself stays agnostic of the underlying mesh—but that separation is also where the first gotchas appear.

Routing and Traffic Shifting

The data plane proxy intercepts every outbound and inbound connection using iptables or IPVS rules installed by the mesh runtime. These rules redirect traffic destined for a clusterIP or service DNS name to a sidecar listening on a local admin port (commonly 15001 for configuration and 15020 for metrics). You can verify the redirection with:

iptables -t nat -L -n | grep 15001

If the rule is absent, the proxy may not have been injected, or the kube-proxy configuration is conflicting with mesh rules.

To test that traffic is actually passing through the proxy, exec into a pod and curl the service through localhost:

kubectl exec -it <pod> --namespace <ns> -- curl -s http://127.0.0.1:15020/health

A 200 OK response indicates the proxy is listening and forwarding. If you receive connection refused, the sidecar may have crashed, or the iptables rule may be redirecting to a port where no process is bound.

Traffic policy—such as percentage-based splitting between two service versions—is expressed in the mesh’s control-plane configuration (typically a CRD or declarative manifest). The control plane propagates the policy to all sidecars; each proxy updates its in-memory routing table and begins honoring the new split after a short convergence period. You can observe the active split by querying the proxy’s stats endpoint:

curl http://127.0.0.1:15020/metrics | grep request_routes

If the metrics show no request_routes labels or all traffic funnels to a single destination, the policy may not have propagated, or a proxy restart may be required.

Observability: Metrics, Logs, and Traces

Most data plane proxies expose a /metrics endpoint on 127.0.0.1:15020. Standard Prometheus scrapes can collect counters for incoming_requests_total, outgoing_requests_total, request_duration_seconds, and grpc_status_code histograms. A typical scrape URL looks like:

curl http://<pod-ip>:15020/metrics

If the response body is empty or returns HTTP 404, the proxy may not have started the admin server, or the port is blocked by a network policy or security group.

Log output from the sidecar is usually sent to stdout, captured by the container runtime, and forwarded to a central logging system (e.g., Elasticsearch, Loki). Look for entries containing proxy started, config loaded, or xds connection established. A missing “proxy started” line within the first 30 seconds of container launch often signals a failed injection or a version mismatch between the control plane and the sidecar binary.

Tracing is enabled by configuring the proxy to export spans to a collector (Jaeger, Zipkin, or a commercial backend). The mesh typically injects a JAEGER_ENDPOINT environment variable or a corresponding config map key. To verify tracing is active, sample a request and check the collector’s UI for a span with the correct trace_id. If spans never appear, confirm that the proxy’s trace_exporter field matches the collector’s expected protocol (OTLP, protobuf, etc.) and that network egress to the collector’s port is allowed.

Policy: mTLS and Access Control

Service meshes commonly provide mutual TLS without application changes. The mesh operates its own certificate authority (often built on Vault or a bundled CA) and issues short-lived per-service identities. When two services establish a connection, each sidecar presents its issued client certificate; the peer verifies it against the mesh’s root CA.

If mTLS fails, the proxy logs an error resembling:

tls: failed to verify certificate: signed by unknown authority

This typically means the peer’s certificate was issued by a different CA, or the mesh’s CA bundle on the receiving side does not include the sender’s issuer. Resolve by ensuring both sides reference the same ca-cert config map, or by rotating the mesh CA and re-injecting sidecars.

Access-control policies are expressed as allow/deny rules matching source/destination service identities, namespaces, or HTTP paths. A typical policy manifest might look like:

apiVersion: security.mesh.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: default
spec:
  selector:
    matchLabels:
      app: backend
  action: Allow
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/default/sa/frontend"]
        port: 443

If a request is unexpectedly rejected, describe the policy with:

kubectl get authorizationpolicy <policy-name> -n <namespace> -o yaml

Check that the selector.matchLabels align with the target service’s labels, and that the principals field correctly references the issuing principal format for your cluster’s authentication backend. A mismatch between the principal format and the token presented by the sidecar will cause the proxy to drop the connection without a clear log entry beyond the TLS handshake failure.

Sharp Edges

Proxy version skew. When the control plane upgrades before the data plane, sidecars continue running the older binary. New policies may not be recognized, and metrics schemas can change, causing dashboards to show blank panels or misinterpreted counters. The cure is a coordinated rollout: drain pods, update the sidecar image version, then resume traffic.

iptables rule conflicts. If your cluster runs additional network plugins (e.g., Calico, Cilium) that also manage iptables chains, mesh rules can be overwritten or cause packet loss. Symptoms include curl timeouts despite healthy endpoints and iptables -L showing duplicate or conflicting rules. Resolve by ordering mesh installation after other CNI plugins, or by using the mesh’s --exclude-cidr flags to carve out non-interfering address ranges.

Config propagation delay. Policy changes are not instant. The control plane pushes updates via xDS (usually xDS v3) to each proxy, which caches the config for a default TTL (often 30–60 seconds). During that window, some requests may follow the old rule, others the new one, producing non-deterministic behavior. You can shorten the TTL by adjusting the proxy’s config_discovery_timeout flag, but be aware that aggressive short TTLs increase control-plane load.

DNS interception failures. Meshes often hijack cluster DNS to return the sidecar VIP instead of the actual service IP. If nslookup <service> returns an unexpected address, or dig +short shows the clusterIP but curl connects to a different host, the mesh’s DNS interceptor may be disabled or overridden by a ClusterPolicy or LocalDNS configuration. Verify that no dnsPolicy: ClusterFirstWithNetInfo or equivalent is suppressing the mesh’s redirect.

Certificate rotation storms. Rotating the mesh CA invalidates all existing service identities. All sidecars must re-establish TLS connections using new certs. If rotation is poorly sequenced, you may see a burst of connection reset by peer errors across the mesh as old connections are abruptly terminated. Plan CA rotations as a controlled rollout, updating the ca-cert config map and restarting a fraction of sidecars before full deployment.

Silent data loss via header manipulation. Some mesh policies rewrite or strip HTTP headers (e.g., X-Request-Id, User-Agent) for routing or security reasons. Applications that depend on those headers may silently misbehave—auth failures, missing audit trails, or incorrect analytics—without any error flag in the mesh logs. Audit policy manifests for header_rewrite or header_remove directives, and validate downstream services against the expected header set after a policy change.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.