Network Observability: Seeing What Flows Through Your Infrastructure

Implement network observability to detect anomalies, debug connectivity issues, and understand traffic patterns. Covers flow logs, packet capture, eBPF-based monitoring, DNS analytics, and the dashboards that make network behavior visible to application teams.

Network observability is the practice of continuously monitoring, understanding, and diagnosing the behavior of data as it flows through a distributed network infrastructure. It matters when network issues are no longer just "outages" but complex, transient, and interdependent—when latency spikes in a microservice aren’t just due to a slow database, but are caused by a misconfigured load balancer, a dropped BGP route, or a burst of UDP packets from a video streaming edge.

Understanding Traffic Flow

To see what flows through your infrastructure, start by capturing packet-level data at key points: edge routers, load balancers, service boundaries, and ingress controllers.

Use tcpdump to capture traffic at the network interface level. For example:

sudo tcpdump -i eth0 -s 0 -w /var/capture/traffic.pcap 'host 192.168.1.100 and port 8080'

This command captures all packets to and from 192.168.1.100 on port 8080, with full packet length (-s 0) and writes the raw data to a file. The resulting .pcap file can be analyzed with Wireshark, tshark, or integrated into a central observability pipeline.

For real-time visibility, stream captures with tcpdump directly to a log aggregator:

sudo tcpdump -i eth0 -n -q -tttt -s 0 -w - 'port 443' | \
  while read line; do
    echo "$(date -Iseconds) $line" | jq -R -c '{timestamp: ., event: "tcpdump", payload: .}'
  done | nc 10.0.0.10:5000

This pipeline outputs each packet as a JSON log line with timestamps, enriched with metadata, and sends it to a remote nc-based log server on port 5000.

Correlating Events Across Network Layers

A packet capture is only the beginning. To understand *why* traffic behaves as it does, correlate events across layers: from physical links to transport, application, and service boundaries.

Use traceroute with precise timing and path analysis:

traceroute -m 30 -q 3 -w 5 -n -I 192.168.1.1

This yields a detailed, stable view of the path from source to destination, including per-hop RTT and packet loss. When combined with ping logs and tcpdump captures, you can correlate a sudden 100ms latency spike in traceroute with a burst of retransmissions in tcpdump.

Using BGP and Routing Data for Visibility

BGP routing changes are invisible without explicit monitoring. A route flapping from one AS to another may cause transient packet loss, yet the core network appears stable.

Use bgpmon to monitor BGP state changes:

bgpmon -f /etc/bgpmon.conf --poll-interval 10s --output-format json

The configuration file defines:

{
  "peers": [
    {
      "address": "10.0.1.1",
      "local_as": 65001,
      "remote_as": 65002,
      "monitor": true,
      "events": [
        "up", "down", "flap", "route_change"
      ]
    }
  ],
  "output": {
    "type": "json",
    "destination": "syslog:10.0.0.10:514",
    "format": "timestamped"
  }
}

This setup emits structured JSON events on BGP state changes, which can be ingested into a metrics or log system. When a peer goes down, the output includes timestamp, peer, event_type, old_state, new_state, and reason.

Monitoring Service-to-Service Traffic

In a service mesh or microservices architecture, traffic between services is the lifeblood. Use istio-proxy logs and envoy stats to expose behavior.

To extract connection and request-level data, enable envoy access logs in the istio-ingressgateway:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: http-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "api.example.com"
      params:
        accessLog:
          format: |
            {
              "timestamp": "%START_TIME%",
              "request_id": "%REQ(X-Request-Id)%",
              "upstream_host": "%UPSTREAM_HOST%",
              "upstream_cluster": "%UPSTREAM_CLUSTER%",
              "response_code": "%RESPONSE_CODE%",
              "duration": "%DURATION%",
              "bytes_received": "%BYTES_RECEIVED%",
              "bytes_sent": "%BYTES_SENT%",
              "path": "%REQ(:path)%",
              "method": "%REQ(:method)%"
            }

Each log entry is a JSON object with fields like timestamp, request_id, and duration. These logs are streamed to a central system via fluentd or vector:


[sources.incoming_logs]
  type = "tcp"
  address = "0.0.0.0:8888"

[sinks.elasticsearch]
  type = "elasticsearch"
  inputs = ["incoming_logs"]
  endpoint = "https://es.example.com:9200"
  index = "istio-access-logs-{+year}-{+month}-{+day}"
  bulk = { max_size_bytes = 1048576, timeout_secs = 30 }

With these logs, you can query for all requests from a service, analyze request latency by path, or detect outlier flows using request_id as a trace ID.

Diagnosing Latency and Loss

When users report slow APIs, you must distinguish between network latency, server processing time, and transport-layer retransmissions.

Use ping with custom intervals and packet sizes to measure round-trip time and packet loss over time:

ping -i 0.5 -c 1000 -s 1472 -W 2 -q 192.168.1.100

The output includes min/avg/max RTT, packet loss percentage, and jitter. When this output is collected over 24 hours, it becomes a time-series of network health.

For deeper analysis, combine ping with mtr (My Traceroute):

mtr -c 100 -i 1 -r -w -z -n 192.168.1.100

The output shows how RTT and packet loss evolve over time across each hop. A sudden increase in loss at hop 5 indicates a problem in the upstream network segment.

Detecting and Responding to Flow Anomalies

A network is not just a collection of links—it is a dynamic flow of packets, streams, and sessions. Observability must detect anomalies in real time.

Use eBPF with bpftrace to instrument network flows. For example, to detect TCP connection bursts:

bpftrace -e '
tracepoint:tcp:tcp_connect
{
  @connections[comm, pid] = count();
}

tracepoint:tcp:tcp_retransmit_skb
{
  @retransmits[comm, pid] = count();
}

interval:10s
{
  println("TCP Connection Summary:");
  foreach (@connections, |k, v|
    printf("%-16s %d\n", k, v)
  );
  println("Retransmission Summary:");
  foreach (@retransmits, |k, v|
    printf("%-16s %d\n", k, v)
  );
  reset(@connections);
  reset(@retransmits);
}'

This script counts new TCP connections and retransmissions every 10 seconds. It outputs a summary table showing which processes are opening many connections, and which are experiencing packet loss.

When combined with tcpdump and BGP data, it becomes possible to correlate a surge in kubernetes-apiserver connections with a BGP route change that brought traffic to a new data center.

Debugging Packet Loss and Reordering

Packet loss is not binary—it can be random, bursty, or correlated with specific flows.

Use iperf3 to generate controlled traffic and measure throughput and loss:

iperf3 -c 192.168.1.100 -t 60 -i 10 -u -b 100M -l 1472

The output includes bandwidth, jitter, and packet loss per interval. When combined with ping and mtr, you can determine whether loss is due to congestion, link errors, or buffer overflows.

For reordering detection, use iperf3 with --reorder and --reorder-ratio:

iperf3 -c 192.168.1.100 -t 30 -u -b 50M -l 1024 \
  --reorder 10 --reorder-ratio 0.1

This simulates a network where 10% of packets are reordered, with a 10% chance of reordering per packet. The result shows how applications behave under reordering—useful for tuning TCP window sizes or for diagnosing issues in distributed databases.

Building a Unified View

Observability is not just about tools—it is about unifying data into a coherent picture.

Use prometheus to collect metrics from network devices and services:

# prometheus.yml
scrape_configs:
  - job_name: 'network'
    static_configs:
      - targets: ['192.168.1.1:9100', '192.168.1.2:9100']
    metrics_path: '/metrics'
    scheme: 'http'
    relabel_configs:
      - source_labels: ['__address__']
        target_label: 'instance'
  - job_name: 'tcp-traceroute'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['10.0.0.10:8080']
    params:
      'query': ['avg(rate(tcp_connections_total[5m]))']

This config scrapes metrics from network exporters (like node_exporter, snmp_exporter) and application-level metrics from a tcp-traceroute service. The resulting time-series data can be visualized in Grafana or used for alerting.

Alert on anomalies using Alertmanager:

route:
  receiver: 'slack'
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
receivers:
  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/ABC123/XYZ789'
        channel: '#network-observability'
        title: '{{ template "slack.title" . }}'
        text: '{{ template "slack.text" . }}'
templates:
  - 'templates/*.tmpl'

An alert is triggered when:

Each alert includes a structured message with links to the relevant data—tracebacks, packet captures, and service logs.

Final Touches: What Fails Silently

  tcpdump -i eth0 -B 1024 -W 10 -C 100 -s 0 -w /var/capture/traffic.pcap

-B 1024: buffer size of 1024 KB, -W 10: rotate 10 files, -C 100: rotate at 100 MB.

  tshark -r traffic.pcap -Y 'tcp.analysis.retransmission' -T fields \
    -e frame.number -e tcp.srcport -e tcp.dstport -e tcp.seq -e tcp.len
  tshark -r traffic.pcap -Y 'udp' -T fields -e udp.length -e udp.port -e frame.time_epoch

These tools, configurations, and patterns form a robust foundation for network observability: a way to see, understand, and act on every packet that flows through your infrastructure.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.