Container Networking Deep Dive: From Pods to Service Mesh

Understand container networking from first principles — how pods communicate, how Kubernetes services route traffic, how network policies enforce security, and when a service mesh is worth the operational complexity.

Container networking spans from the pod-level connectivity within a cluster to the end-to-end service mesh that orchestrates microservices across environments. This reference page details the mechanics, configurations, and failure modes of container networking, from the moment a container boots inside a pod to the orchestration of traffic across a service mesh. It is intended for engineers who must debug, configure, or extend containerized systems, particularly when a service fails across multiple clusters or when latency spikes appear in a distributed application.

Pod Networking: The Foundation of Container Communication

Containers within a pod share a network namespace, enabling them to communicate via localhost and access shared volumes. Each pod receives an IP address from the cluster’s IP address pool, typically assigned by the CNI (Container Network Interface) plugin.

Pod IP Assignment and CNI Integration

When a pod is scheduled to a node, the CNI plugin—such as Calico, Cilium, or Flannel—configures the pod’s network interface. The pod’s eth0 interface is assigned an IP from the cluster’s subnet, and routes are installed on the host.


apiVersion: v1
kind: Pod
metadata:
  name: app-pod
  labels:
    app: frontend
spec:
  containers:
  - name: app
    image: nginx:alpine
    ports:
    - containerPort: 8080
  - name: sidecar
    image: redis:alpine
    ports:
    - containerPort: 6379

The CNI plugin populates the pod’s eth0 with an IP via a CNIConfig file in /etc/cni/net.d/. The network configuration is a JSON object passed from the kubelet to the CNI plugin during pod creation.

{
  "cniVersion": "0.4.0",
  "name": "pod-network",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cbr0",
      "isDefaultGateway": true,
      "ipMasq": true,
      "hairpinMode": true,
      "ipam": {
        "type": "host-local",
        "subnet": "10.244.1.0/24",
        "routes": [
          {
            "dst": "0.0.0.0/0"
          }
        ]
      }
    }
  ]
}

The host-local IPAM plugin allocates IPs from a pre-defined pool, and the bridge plugin creates a Linux bridge (cbr0) on the node, attaching the pod’s eth0 to it.

Common Failure Modes

Service Discovery and Load Balancing: Exposing Pods

A Kubernetes Service exposes a set of pods as a single endpoint. Services use a virtual IP (ClusterIP) and are backed by endpoints that list all pod IPs.

Service Configuration and Endpoints

apiVersion: v1
kind: Service
metadata:
  name: frontend-service
  labels:
    app: frontend
spec:
  selector:
    app: frontend
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  type: ClusterIP

When the service is created, Kubernetes creates an Endpoints object that maps the service IP to the list of pod IPs.

apiVersion: v1
kind: Endpoints
metadata:
  name: frontend-service
subsets:
  - addresses:
      - ip: 10.244.1.5
        targetRef:
          kind: Pod
          name: app-pod
          namespace: default
    ports:
      - port: 80
        protocol: TCP
        name: http

The kube-proxy daemon (running on each node) synchronizes the service’s ClusterIP with iptables or IPVS rules.

iptables Mode Configuration

In iptables mode, kube-proxy generates rules that redirect traffic from the service IP to the pod IPs.

# View generated iptables rules
sudo iptables -L -n | grep 10.100.100.100
# Output:
# -A KUBE-SERVICES -d 10.100.100.100/32 -p tcp -m tcp --dport 80 -j KUBE-SVC-ABCD1234
# -A KUBE-SVC-ABCD1234 -m statistic --mode random --probability 0.5 -j KUBE-SEP-1234ABCD
# -A KUBE-SEP-1234ABCD -s 10.244.1.5 -j DNAT --to-destination 10.244.1.5:8080

The KUBE-SVC-ABCD1234 chain loads balances across the KUBE-SEP-... chains, each handling one pod.

Service Failures

Ingress: Routing External Traffic to Services

External traffic enters the cluster via an Ingress resource, which routes HTTP(S) requests to services based on hostnames and paths.

Ingress Configuration with Rules

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: api-service
                port:
                  number: 8080
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend-service
                port:
                  number: 80

The ingress controller—such as NGINX, Traefik, or Istio Ingress Gateway—processes the Ingress object and generates configuration for a reverse proxy.

NGINX Ingress Controller Configuration

The controller watches for Ingress resources and generates nginx.conf dynamically.

# Generated nginx.conf fragment
upstream app-example-com-api {
  server 10.244.1.10:8080 max_fails=3 fail_timeout=30s;
}

server {
  listen 80;
  server_name app.example.com;

  location /api {
    rewrite ^/api(.*)$ $1 break;
    proxy_pass http://app-example-com-api;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
  }

  location / {
    proxy_pass http://frontend-service;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
  }
}

Ingress Pitfalls

Service Mesh: Observability and Control Plane

A service mesh—such as Istio, Linkerd, or Consul—adds a data plane of sidecar proxies (typically Envoy) to every pod and a control plane for configuration and telemetry.

Sidecar Injection and Traffic Management

When a pod is created, the service mesh’s mutating admission controller injects a sidecar container with the data plane proxy.

apiVersion: v1
kind: Pod
metadata:
  name: user-service
  annotations:
    sidecar.istio.io/inject: "true"
spec:
  containers:
  - name: user-service
    image: user-service:v1
    ports:
    - containerPort: 8080
  - name: istio-proxy
    image: istio/proxyv2:1.18
    args:
      - proxy
      - sidecar
      - --configPath
      - /etc/istio/proxy
      - --binaryPath
      - /usr/local/bin/envoy
      - --serviceCluster
      - user-service
      - --serviceNode
      - pod-12345.user-service
      - --serviceUID
      - kubernetes://pod/user-service
    ports:
      - containerPort: 15000
        name: http-envoy
      - containerPort: 15001
        name: admin

The proxy listens on 15000 and forwards traffic from the application container on 8080 to the network.

Traffic Routing with Istio VirtualService

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: user-service-virtual
spec:
  hosts:
    - user.example.com
  http:
    - match:
        - uri:
            prefix: /v1
      route:
        - destination:
            host: user-service
            subset: v1
          weight: 70
        - destination:
            host: user-service
            subset: v2
          weight: 30
    - match:
        - uri:
            prefix: /v2
      route:
        - destination:
            host: user-service
            subset: v2

This config routes 70% of traffic to v1 and 30% to v2, with v2 handling all /v2 requests.

Service Mesh Failure Modes

Inter-Cluster Networking: Multi-Cluster Service Mesh

For applications spanning multiple clusters, service mesh control planes coordinate traffic across clusters.

Cross-Cluster Service Discovery

A MeshService object defines a service that spans multiple clusters.

apiVersion: networking.istio.io/v1beta1
kind: MeshService
metadata:
  name: user-service
  namespace: istio-system
spec:
  endpoints:
    - address: 10.244.1.5
      targetCluster: cluster-a
      labels:
        version: v1
    - address: 10.244.2.8
      targetCluster: cluster-b
      labels:
        version: v2

The control plane maintains a global view of all services and routes traffic using a global load balancer or service mesh gateway.

Failure in Multi-Cluster Setup

Summary of Key Concepts

Each layer builds on the previous: a pod’s IP is the foundation, services expose pods, ingress routes external traffic, and the service mesh adds observability and control. Failures cascade: a missing IP breaks service discovery, which breaks ingress routing, which breaks service mesh traffic.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.