Network Observability Tools

Network observability is the practice of inferring the state of a network stack from external signals—packet traces, interface counters, flow metrics—when direct inspection is impractical or prohibited. It matters when a service fails to handshake across subnets, when latency jumps without a clear bottleneck, when packets are dropped silently by a firewall or misconfigured qdisc, or when you're debugging a distributed system that spans bare‑metal, virtualization, and containers. This reference collects the concrete commands and patterns you can run today to generate those signals, organized by the specific question you're trying to answer.

Capturing Live Traffic on a Host

tcpdump is the de facto standard for real‑time packet capture on Unix‑like systems. The most common invocation for immediate, name‑resolution‑free output is:

tcpdump -i en0 -nn -s 0 -w -

• -i en0 selects the interface. On Linux replace with eth0, ens3, or the output of ip link show.
• -nn prints numeric addresses only, skipping DNS lookups so the command returns instantly.
• -s 0 sets the snapshot length to the full packet size; without it tcpdump truncates to 96 bytes by default, which can drop critical headers.
• -w - writes raw packets to stdout; pipe to tshark or gzip for storage.

If the interface name is wrong you’ll get:

tcpdump: en0: No such device

If you lack the cap_net_raw capability the process exits silently with status 1 and no output—hence the frequent habit of prefixing with sudo or checking setcap cap_net_raw+ep tcpdump.

tshark (Wireshark’s CLI counterpart) lets you apply display filters at capture time:

tshark -i eth0 -Y "tcp.flags.syn == 1 and port 443" -c 50

• -Y is a display filter applied after capture (unlike -f which is a pcap filter expression).
• -c 50 stops after 50 packets, useful for quick sanity checks without filling disks.

ngrep filters on payload content without needing the full GUI:

ngrep -d eth0 "GET /api" host 203.0.113.45

• -d eth0 picks the interface.
• The pattern is a POSIX extended regular expression; ngrep will match it against the payload of passing packets.
• Unlike tcpdump’s -A/-X hex dump, ngrep shows the actual application‑layer string, which speeds up spotting malformed requests or leaked secrets.

Common pitfall: ngrep on a high‑throughput interface can drop packets if the regex engine blocks; on a 10 Gbps link limit captures to a single flow or use a more specific filter.

Tracing Path and Measuring Latency

mtr combines traceroute and ping in a single looping session. The canonical form for a quick route health check is:

mtr -r -c 10 1.1.1.1

• -r runs once and exits (non‑repetitive mode). Without it mtr loops until you quit with q.
• -c 10 sends 10 probe packets per hop. Adjust up for noisy paths, down for speed.
• Output shows loss percentage, SRTT, and the final destination’s address. If a hop is marked ?, the probe was dropped—often due to firewalls rate‑limiting ICMP or a missing reverse‑path filter.

traceroute with TCP SYN instead of ICMP is useful when ICMP is blocked across a corporate boundary:

traceroute -T -p 443 -n 8.8.8.8

• -T forces a TCP SYN probe (default is ICMP UDP). Many firewalls allow outbound TCP 443 even when ICMP is dropped.
• -n suppresses DNS resolution, returning IP addresses only—critical when your DNS is the very thing you’re troubleshooting.
• If -T is unsupported (some older traceroute builds), you’ll see traceroute: cannot resolve host or the tool falls back to UDP probes automatically; the exit status will be 2 and the output will list only the hops reachable via UDP.

When both ICMP and TCP probes fail, the output terminates early with:

traceroute: reached maximum hops (30) without response

Inspecting Interface Statistics and Queue Discipline

ethtool exposes driver‑ and hardware‑level counters that ip link alone won’t show. To view all statistics on an interface:

ethtool -S eth0

Typical output lines you’ll care about:

• rx_packets, tx_packets
• rx_errors, tx_errors
• rx_dropped, tx_dropped
• collisions, rx_crc_errors, rx_frame_errors
• rx_missed_errors (packets lost because the host CPU couldn’t ingest them fast enough)

If the driver doesn’t expose a counter, the line reads 0 and ethtool may print:

ethtool -S eth0: Cannot get statistics: Operation not supported

This commonly happens on virtual NICs in certain cloud VM types or on container‑level vNICs where the host kernel doesn’t forward the stats.

ip from iproute2 complements ethtool with queue‑ing discipline info:

ip -s link show dev eth0

Output includes txqueuelen 1000 (default transmit queue length) and rx_queue/tx_queue counters if your kernel supports it. A suddenly full transmit queue (txqueuelen hit) is a classic symptom of a burst‑y producer without back‑pressure, often misdiagnosed as a “network outage” when the real fix is increasing the queue or enabling ECN.

Debugging Container and Pod Network Issues

In Kubernetes the fastest way to verify that a pod can reach an external endpoint is:

kubectl exec -it <pod> -- nslookup <service>.namespace.svc.cluster.local

• If the pod uses ClusterFirst DNS, the query should resolve to the cluster IP. If you get server refused, the pod’s DNS config is misaligned or the CoreDNS service is down.
• If the DNS query succeeds but the service is unreachable, fall back to a raw TCP check:

kubectl exec -it <pod> -- curl -s -o /dev/null -w "%{http_code} %{time_total}\n" https://example.com

• -s silences progress, -o /dev/null discards body, -w prints HTTP status and total time.
• A 000 status usually means the TCP handshake was refused or the TLS handshake failed inside the container.

If the pod runs a CNI plugin that manipulates iptables, you may see iptables: No such file or directory when trying to list rules from inside the pod. That’s not a network outage—it’s a missing kernel module or a sandboxed runtime (gVisor, Kata Containers) that virtualizes the network stack.

Correlating Packets with Logs and Traces

The most reproducible local workflow is to capture a short trace while reproducing the failure, then grep the payload for correlating IDs:

tcpdump -i en0 -s 0 -w /tmp/capture.pcap "port 8080" &
PID=$!

curl -v https://api.example.com/v1/resource > /tmp/curl.out 2>&1
wait $PID
tshark -r /tmp/capture.pcap -Y "http.request.uri contains \"/v1/resource\"" -T fields -e frame.time -e ip.src -e ip.dst -e http.request.method

• Backgrounding tcpdump (&) prevents it from blocking the reproducing command.
• The tshark field extraction shows timestamp, endpoints, and method without the full HTTP body.
• To match with application logs, include a request‑ID header in both the curl and your app’s log output (e.g., X-Request-Id: $UUID). Then grep across both streams.

If you’re on a host where tcpdump requires BPF filtering and your shell lacks BPF privileges, you’ll see:

tcpdump: BPF filter compilation failed: Operation not permitted

In that case, drop to root or use a user‑namespace‑aware variant (unshare --net tcpdump ...), or switch to tshark running as the capture group member.

Sharp Edges: What Silently Fails, Confused Options, Production Lessons

Permission‑silent failures – tcpdump and tshark exit with status 1 and no output when run without cap_net_raw on a non‑promiscuous interface. The error message is easy to miss in automated scripts that only check stdout. Always strace the first line or sudo the binary in CI pipelines.

Filter syntax drift – tcpdump uses BPF syntax (host 10.0.0.1 and port 80). tshark extends this with libpcap’s full expression parser but also accepts Wireshark‑style display filters after -Y. ngrep uses its own mini‑regex dialect. A filter that works in one will silently produce zero matches in another. When porting a capture pipeline between tools, rewrite the expression rather than copy‑pasting.

mtr ICMP vs TCP – On paths where firewalls rate‑limit ICMP source‑quench or type‑3 codes, mtr -r -c 10 will report 100 % loss on those hops, leading operators to assume a broken route. Adding -T (TCP SYN probes) or --report with --tcp flag forces mtr to use TCP, but this requires the destination port to be open and may not reflect ICMP‑based QoS policies.

ethtool statistics reset – Some NIC drivers zero‑out counters on every ethtool -S invocation if the driver is in a “reset” state after a link flap. If you’re tracking drop rates over time, capture the first reading, wait a known interval, then subtract. Failing to account for this rollover makes trend analysis misleading.

kubectl exec DNS caveats – The nslookup command inside a pod uses the pod’s /etc/resolv.conf. If that file points to a ClusterIP CoreDNS service that itself is experiencing latency, the lookup inside the pod will appear slow even though the external DNS is fine. Always verify the host’s /etc/resolv.conf independently, or use dig +short against a known external resolver (8.8.8.8) from within the pod to isolate the source.

curl -w format gotchas – The -w flag’s format specifiers are not documented in the curl man page on all distributions; some embed %{time_total} as wall‑clock time, others as CPU time. The portable subset is %{http_code}, %{time_appconnect}, %{time_connect}, %{time_starttransfer}. Stick to the ones your version reports via curl --version; otherwise you may log a time value that looks reasonable but is actually the time spent in DNS resolution only.

Packet capture disk I/O – Writing raw pcap files with tcpdump -w on a high‑throughput interface can saturate the local filesystem, especially on root partitions or ephemeral SSD nodes in cloud VMs. The process may block or the kernel may drop packets due to write latency. Pipe to gzip or split in real time:

tcpdump -i en0 -s 0 -w - | gzip > /tmp/capture_$(date +%s).pcap.gz

or use tshark’s -b filesize:1000 rotation flag to keep files bounded.

Traceroute port probing side‑effects – traceroute -T -p 443 opens a TCP SYN to each hop’s port 443. If a middlebox performs strict connection tracking, it may record a half‑open connection per hop, eventually exhausting connection‑track tables on a stateful firewall. In production environments, prefer -T only on a dedicated test host or limit probes with -f (first hop) and -m (max hops).

These edges aren’t bugs—they’re expected behavior given the kernel‑level interfaces these tools wrap. Knowing the exact condition that triggers each symptom lets you distinguish a configuration mistake from a platform limitation, and stop chasing the wrong root cause.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.