Network Troubleshooting

Production engineering guide for network troubleshooting covering patterns, implementation strategies, and operational best practices.

Network troubleshooting is the practice of diagnosing and resolving connectivity issues across networks, from local subnets to globally distributed systems. It matters when a service is unreachable, latency is high, packets are dropped, or a client reports intermittent failures—especially when the problem spans multiple layers, services, and geographic regions.

Diagnose a Service Unreachable

When a client cannot reach a service, begin with basic reachability checks using ping and traceroute.

ping -c 4 -W 2 example.com

If ping fails, use traceroute to identify where the path breaks:

traceroute -m 30 -q 3 -w 5 example.com

A failure at a hop indicates a network segment failure. A consistent increase in latency or packet loss at a specific hop suggests congestion, routing issues, or a bottlenecking device.

Check the network stack using tcpdump on the server:

tcpdump -i eth0 -n -s 0 -v -w /tmp/traceroute.pcap 'host example.com'

Examine the packet captures with tshark:

tshark -r /tmp/traceroute.pcap -T fields -e frame.number -e ip.src -e ip.dst -e tcp.flags -e tcp.window_size

Look for:

Identify High Latency

High latency manifests as slow response times, especially in stateful services or real-time applications. Use mtr (My Traceroute) for real-time path analysis:

mtr -c 100 -r -l -i 1 example.com

Watch for:

Use curl to measure end-to-end HTTP latency:

curl -w "@latency-format.txt" -o /dev/null -s http://example.com

Where latency-format.txt contains:

time_namelookup:  %{time_namelookup}\n
time_connect:      %{time_connect}\n
time_appconnect:   %{time_appconnect}\n
time_pretransfer:  %{time_pretransfer}\n
time_redirect:     %{time_redirect}\n
time_starttransfer: %{time_starttransfer}\n
time_total:        %{time_total}\n

This reveals:

A high time_appconnect suggests TLS handshake issues—check for:

Detect Packet Loss and Jitter

Packet loss and jitter are common in video, voice, and real-time applications. Use ping with large payloads and burst mode:

ping -c 100 -s 1472 -i 0.1 -W 3 example.com

Examine the output:

Use iperf3 to measure bandwidth, loss, and jitter under sustained load:

iperf3 -c example.com -t 60 -i 1 -u -b 1M -l 1472

Key metrics:

Troubleshoot DNS Resolution Failures

When clients cannot resolve hostnames, inspect DNS query behavior using dig and nslookup.

dig +time=5 +retry=3 +dnssec example.com

Check:

Use systemd-resolve to inspect local DNS resolver state:

resolvectl status

Look for:

Check systemd-resolved logs:

journalctl -u systemd-resolved.service -f

Common issues:

Use tcpdump to capture DNS queries:

tcpdump -i any -n -s 0 -v -w /tmp/dns.pcap 'port 53'

Then analyze with tshark:

tshark -r /tmp/dns.pcap -T fields -e dns.qry.name -e dns.qry.type -e dns.aa -e dns.ad -e dns.rcode

Look for:

Diagnose TCP Connection Problems

When a service is up but slow or intermittent, examine TCP connection state using ss and netstat.

ss -tuln -o state established

Inspect:

Check for:

Use tcpdump to capture handshake and data flow:

tcpdump -i eth0 -n -s 0 -v -w /tmp/tcp-handshake.pcap 'tcp port 8080 and (tcp[13] & 0x02 != 0) or (tcp[13] & 0x01 != 0)'

Analyze:

Check TCP options:

Use ethtool to inspect interface-level TCP settings:

ethtool -c eth0

Look for:

Validate End-to-End Path and Service Health

Use curl to probe service health with realistic payloads:

curl -H "Accept: application/json" \
     -H "Content-Type: application/json" \
     -d '{"id": 123, "name": "test"}' \
     -X POST http://example.com/api/v1/users \
     -w "@health-check-format.txt" \
     -o /dev/null -s

Include a health-check-format.txt file:

time_namelookup:  %{time_namelookup}\n
time_connect:      %{time_connect}\n
time_appconnect:   %{time_appconnect}\n
time_pretransfer:  %{time_pretransfer}\n
time_starttransfer: %{time_starttransfer}\n
time_total:        %{time_total}\n
http_code:         %{http_code}\n

Check:

Use kubectl to check service reachability in Kubernetes:

kubectl get pods -l app=api-server -o wide
kubectl describe pod <pod-name>
kubectl exec <pod-name> -- curl -s http://localhost:8080/healthz

Inspect:

Use systemctl to verify service status and logs:

systemctl status api-server.service
journalctl -u api-server.service --since "1 hour ago" --follow

Look for:

Common Pitfalls and Silent Failures

These techniques form a coherent workflow: start with reachability, drill into latency and packet loss, validate DNS and TCP, and close with end-to-end service checks. Each step builds on the previous, revealing hidden failures that only surface under real load.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.