Service Mesh Architecture | The Garnet Wiki

Service mesh architecture provides a transparent, programmable layer for managing service-to-service communication in distributed systems. It is essential when services are deployed across multiple clusters, regions, or cloud providers, and when you need consistent observability, reliability, and security across a microservices ecosystem. It matters when latency, failure, and configuration drift between services start to dominate system behavior.

Core Patterns of Service Mesh Operation

Interception and Proxy Injection

A service mesh operates by inserting a sidecar proxy—typically Envoy, Istio’s data plane, or a custom deployment of the same—into the network path of every service instance. This proxy intercepts all incoming and outgoing traffic, enabling features like load balancing, retries, circuit breaking, and mTLS without service code changes.

The proxy is injected at deployment time via a controller. In Kubernetes, this is typically done using a mutating admission webhook that patches the pod spec with a sidecar container and configures the service’s initContainer to start the proxy.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: product-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: product-service
  template:
    metadata:
      labels:
        app: product-service
    spec:
      containers:
      - name: product-service
        image: example.com/product-service:v1.2
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
      - name: istio-proxy
        image: istio/proxyv2:1.19
        args:
        - proxy
        - --configPath
        - /etc/istio/proxy
        - --binaryPath
        - /usr/local/bin/envoy
        - --serviceCluster
        - product-service
        - --serviceNode
        - cluster1.product-service
        ports:
        - containerPort: 15090
          name: stats
        - containerPort: 15020
          name: http-envoy-admin
        - containerPort: 15000
          name: http-envoy-metrics
        - containerPort: 15001
          name: tcp-envoy-metrics
        volumeMounts:
        - name: istio-envoy-config
          mountPath: /etc/istio/proxy
        - name: istio-certs
          mountPath: /etc/certs
        - name: istio-data
          mountPath: /var/lib/istio
      volumes:
      - name: istio-envoy-config
        configMap:
          name: istio-proxy-config
      - name: istio-certs
        secret:
          secretName: istio-certs
      - name: istio-data
        emptyDir: {}

The sidecar is configured via a ProxyConfig object that defines its behavior. Key fields include:

clusterName: the logical cluster name for routing.
enableSidecarInjector: whether to enable automatic injection.
enablePrometheusMetrics: enables metrics export to Prometheus.
controlPlaneAuthPolicy: sets mTLS policy (none, mutual, strict).
connectTimeout: duration for connection setup, e.g., 10s.

When a service starts, it registers with the control plane via a gRPC stream. The control plane updates the proxy’s configuration dynamically by pushing updates to the Envoy configuration over xDS (Discovery Service).

Traffic Management and Routing

Traffic shaping is the primary responsibility of a service mesh. It enables canary deployments, A/B testing, and fault injection.

A VirtualService defines how traffic is routed to services. It uses a declarative format that maps hostnames, paths, and headers to destination services with weights, retries, and timeouts.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: product-service
spec:
  hosts:
  - product.example.com
  http:
  - match:
    - uri:
        prefix: /api/v1/products
    - headers:
        user-agent:
          regex: ".*mobile.*"
    route:
    - destination:
        host: product-service
        subset: v1
      weight: 70
    - destination:
        host: product-service
        subset: v2
      weight: 30
    retry:
      attempts: 3
      perTryTimeout: 5s
      retryOn: "5xx,gateway-error,connect-failure"
    timeout: 10s
  - match:
    - uri:
        prefix: /api/v1/products
    route:
    - destination:
        host: product-service
        subset: v3
      weight: 100
    fault:
      delay:
        fixedDelay: 2s
        percentage:
          value: 50
      abort:
        httpStatus: 500
        percentage:
          value: 25

The mesh evaluates route rules in order and applies them to incoming HTTP requests. The subset field refers to a DestinationRule, which defines load balancing policies, connection pool settings, and circuit breaking.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: product-service
spec:
  host: product-service
  trafficPolicy:
    loadBalancer:
      simple: LEAST_CONN
    connectionPool:
      http:
        http1MaxPendingRequests: 100
        maxRequestsPerConnection: 10
        idleTimeout: 30m
      tcp:
        maxConnections: 1000
    outlierDetection:
      consecutive5xxErrors: 5
      baseEjectionTime: 30s
      interval: 10s
      maxEjectionPercent: 50

When a service receives a request, the sidecar applies the routing rules, and the control plane can dynamically update them via configuration updates. The mesh maintains a consistent view of service availability through health checks and service discovery.

Observability and Tracing

A service mesh enables end-to-end observability by collecting metrics, logs, and distributed traces across service boundaries.

Each proxy exposes metrics via Prometheus. Common metrics include:

envoy_cluster_upstream_cx_active: number of active connections.
envoy_http_downstream_cx_total: total upstream connections.
envoy_cluster_http_rq_2xx, envoy_cluster_http_rq_5xx: request success and error rates.
envoy_cluster_upstream_rq_time: latency distribution.

The mesh collects these metrics and enriches them with metadata such as request IDs, service names, and trace IDs.

Tracing is enabled via OpenTelemetry or Zipkin. The control plane configures each proxy to emit traces via a Tracing object.

apiVersion: telemetry.istio.io/v1alpha1
kind: Tracing
metadata:
  name: default-tracing
spec:
  providers:
  - name: zipkin
    zipkin:
      endpoint: http://zipkin-collector:9411/api/v2/spans
      timeout: 10s
      maxPathLength: 100
  - name: jaeger
    jaeger:
      endpoint: http://jaeger-collector:14268/api/spans
      sampling: 0.5
      tags:
      - key: service
        value: "product-service"

When a request flows through multiple services, the mesh propagates the trace context via HTTP headers:

x-request-id: unique ID for the request.
x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, x-b3-flags: used by Zipkin/B3.

The mesh ensures that every hop adds a span to the trace, and that spans are nested hierarchically. The control plane can attach metadata such as service version, deployment environment, and request duration.

Security: mTLS and Policy Enforcement

A service mesh enforces service-to-service security via mutual TLS (mTLS). The control plane issues and manages certificates using a certificate authority (CA).

Each proxy is configured to:

Present a certificate to the peer.
Verify the peer’s certificate.
Enforce mTLS policies.

The mesh uses a PeerAuthentication resource to define the mTLS policy.

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: STRICT
  selector:
    matchLabels:
      app: product-service
  portLevelMtls:
    8080:
      mode: STRICT
    8081:
      mode: PERMISSIVE

In STRICT mode, all traffic must be mTLS-secured. In PERMISSIVE mode, both mTLS and plain TLS are allowed, enabling gradual rollout.

The mesh also supports AuthorizationPolicy to define fine-grained access control.

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: product-service-access
spec:
  selector:
    matchLabels:
      app: product-service
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/default/sa/product-service"]
    - source:
        namespaces: ["prod", "staging"]
    to:
    - operation:
        methods: ["GET", "POST"]
        paths: ["/api/v1/products/*"]
    when:
    - key: request.headers[authorization]
      values: ["Bearer token123"]
    - key: request.headers[x-user-role]
      values: ["admin", "editor"]

The control plane evaluates these policies at the proxy level. The proxy checks each incoming request against the policy and enforces access control decisions.

Failure Modes and Edge Cases

Several silent failure modes occur in service mesh deployments:

Proxy misconfiguration: if the ProxyConfig is not synced, the sidecar may use stale or incomplete settings. The control plane logs ProxyConfigSyncError events, and the proxy logs Failed to load configuration from xDS on startup.

Delayed configuration propagation: when a new VirtualService is applied, it may take several seconds to propagate to all proxies. During this window, traffic may be routed inconsistently. The mesh can detect this via envoy_cluster_upstream_cx_draining, which indicates a proxy is being drained during configuration update.

Certificate rotation and expiry: if the CA is not monitored, expired certificates lead to TLS handshake failed errors in logs. The proxy logs TLS certificate verification failed when the peer’s certificate is not trusted or has expired.

Trace context loss: if a service uses a custom HTTP client that does not propagate B3 headers, traces may be fragmented. The mesh logs Trace context not propagated when x-b3-traceid is missing.

Circuit breaking triggers: when outlierDetection thresholds are exceeded, the proxy stops sending requests to failing instances. The envoy_cluster_outlier_detection_consecutive_5xx_errors metric shows the number of consecutive 5xx errors, and the envoy_cluster_outlier_detection_active flag indicates whether the circuit is open.

Traffic shifting delays: when a VirtualService route weight is updated, the proxy does not immediately apply the new weights. The envoy_cluster_load_assignment updates may be delayed by up to 30 seconds. The mesh logs Route weight update applied when the change is fully active.

Sidecar startup latency: the proxy may start before the main application container, leading to early requests being dropped. The control plane logs Proxy startup lag and the proxy logs Proxy ready but application not yet listening.

Configuration and Deployment Workflow

The service mesh is configured via declarative resources in the control plane. The typical workflow is:

Define DestinationRule, VirtualService, PeerAuthentication, and AuthorizationPolicy in YAML.
Apply the files using kubectl apply -f.
Monitor proxy logs via kubectl logs <pod-name> -c istio-proxy.
Validate routing with curl and http headers:

curl -H "x-b3-traceid: abc123" \
     -H "x-user-role: admin" \
     -H "authorization: Bearer token123" \
     http://product.example.com/api/v1/products

Check metrics with curl http://<proxy-ip>:15090/stats or via Prometheus.

Debug configuration issues with istioctl proxy-config:

istioctl proxy-config cluster product-service-789abc
istioctl proxy-config route product-service-789abc
istioctl proxy-config listener product-service-789abc
istioctl proxy-config bootstrap product-service-789abc

Validate mTLS with openssl s_client -connect product-service:8080 -servername product-service.

The service mesh becomes a single source of truth for service behavior. It enables the team to shift responsibility from application developers to platform engineers, allowing consistent, observable, and resilient service communication across the organization.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.