Network Troubleshooting Methodology

Production-grade guide to network troubleshooting methodology covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.

Network troubleshooting is the disciplined process of diagnosing, isolating, and resolving network issues across systems, services, and infrastructure. It matters when packets fail to reach their destination, applications become unresponsive, or latency spikes without clear cause—when the network is the root of a failure, not just a contributor.

Diagnose the Problem

Begin with a clear picture of what is wrong. Use consistent, repeatable tools to gather symptoms from the edge outward.

Identify Symptoms

Start with high-level observations: application timeouts, increased error rates, service degradation, or user reports. These symptoms must be concrete and measurable.

> Common failure modes: > - Packet loss on a hop: 10.10.10.1 is dropping packets due to buffer overflow or link failure. > - High jitter in the last 5 hops: last-mile or backbone network instability. > - Timeouts in curl but not in ping: application-level issues (e.g., TLS handshake failure, slow server processing).

Confirm the Failure

Define the failure precisely: when did it start? Is it consistent? Is it local or global?

  tcpdump -i eth0 -n -s 0 -w /tmp/troubleshoot.pcap \
    'host example.com and port 80'
  tshark -r /tmp/troubleshoot.pcap -T fields -e frame.time_epoch \
    -e ip.src -e ip.dst -e tcp.flags -e tcp.window_size -e http.request.method

> Silent failure modes: > - A ping response is received, but the ICMP payload is not echoed — the host replies but the packet is not fully processed. > - DNS resolution succeeds, but the TCP handshake fails — the issue is in the transport layer, not the network layer. > - TLS handshake fails with alert level: warning, description: handshake_failure — the server sends a ServerHello but the client does not send ClientKeyExchange.

Isolate the Layer

Use the OSI model to narrow failure to a specific layer. Start from the physical layer and work upward.

Verify that bits are correctly transmitted across links.

> Common layer 1–2 issues: > - rx_crc_errors > 0: bad cabling, faulty NIC, or noise on the link. > - link_status down: physical layer failure (cable, SFP, switch port). > - MTU mismatch: MSS in TCP SYN is smaller than MTU in IP header — path MTU discovery failed. > - Jumbo frames not negotiated: frame size = 9000 on sender but MTU = 1500 on receiver.

Layer 3: Network

Verify IP connectivity, routing, and addressing.

> Key layer 3 failure modes: > - Default route missing: ip route get 1.1.1.1 returns dev lo, src 127.0.0.1 — no external connectivity. > - Multiple gateways with different metrics: routing is not optimal. > - IP address conflict: two hosts have the same IP, leading to ARP flapping. > - Routing loop: ip route get 10.0.0.1 shows via 10.0.0.2 but 10.0.0.2 has gateway 10.0.0.1 — loop detected.

Layer 4: Transport

Diagnose TCP and UDP issues.

> Critical TCP observations: > - High RTO (Retransmission Timeout): RTO = 1.5 * RTT but RTO > 2s — network is unstable. > - Slow start not progressing: cwnd increases by 1 MSS per RTT, but stays at cwnd = 1 for multiple RTTs. > - Fast retransmit triggered: Dup ACKs > 3 for a single segment. > - SACK not used: SACK: 1000, 1500, 2000 missing in TCP options — sender cannot efficiently retransmit lost segments. > - TCP window scaling not negotiated: window size = 65535, but rwnd = 16384 — receiver window is not being fully utilized.

Layer 5–7: Application

Verify end-to-end functionality of services.

> Application-layer failure modes: > - HTTP 400 Bad Request: malformed request body (Content-Type: application/json but {"name": "John"} missing "). > - HTTP 500 Internal Server Error: Server: nginx, Date: Thu, 01 Jan 2025 00:00:00 GMT, Connection: keep-alive, Content-Length: 0 — server returns empty body. > - TLS handshake failure: alert: level: fatal, description: handshake_failure — client and server disagree on cipher suite or certificate chain. > - Cookie not set: Set-Cookie: session=abc123; Path=/; HttpOnly; Secure; Max-Age=3600 — missing Secure or HttpOnly. > - Connection pooling failure: Keep-Alive not negotiated, Connection: keep-alive but Keep-Alive: timeout=60, max=100 — connection closing after 100 requests.

Validate the Fix

After identifying a root cause, apply a fix and verify its impact.

Apply and Verify

  sysctl -w net.ipv4.tcp_window_scaling=1
  sysctl -w net.ipv4.tcp_sack=1
  sysctl -w net.ipv4.tcp_rmem='4096 87380 6291456'
  sysctl -w net.ipv4.tcp_wmem='4096 16384 4194304'
  ss -i -o 'state established' | grep -A 5 -B 5 'cwnd'
  ab -n 10000 -c 100 http://api.example.com/users

> Silent success and failure: > - tcpdump shows SYN, ACK but ss shows TCP state: SYN_RECV — the server received the handshake but did not accept the connection. > - curl returns 200, but httpie shows Transfer-Encoding: chunked and Content-Length: 0 — the server sends chunked data but forgets to send 0\r\n\r\n. > - traceroute shows stable RTT, but ping shows random delays — the issue is in the network path, not the host.

Iterate and Document

Troubleshooting is not a one-off task. Build a repeatable process and shared knowledge.

  ## Network Troubleshooting Checklist

  ### Initial Diagnosis
  - [ ] `ping -c 10 -i 0.5 example.com`
  - [ ] `traceroute -q 3 -m 30 example.com`
  - [ ] `curl -v http://api.example.com/health`

  ### Layer 3 (IP)
  - [ ] `ip route show`
  - [ ] `ip addr show`
  - [ ] `ip -4 route get 1.1.1.1`

  ### Layer 4 (TCP)
  - [ ] `ss -tuln`
  - [ ] `ss -i -o 'state established'`
  - [ ] `tcpdump -i any -n -s 0 -w /tmp/troubleshoot.pcap 'port 80'`

  ### Layer 7 (Application)
  - [ ] `httpie --print=hbH http://api.example.com/health`
  - [ ] `curl -v -H "Accept: application/json" http://api.example.com/users/123`
  - [ ] `tshark -r /tmp/troubleshoot.pcap -T fields -e tcp.flags -e tcp.window_size`

> Common pitfalls in iteration: > - Assuming a fix is complete when it is not: ping shows success, but curl still fails — the fix addressed one layer but not another. > - Overlooking the impact of changes: increasing cwnd but not adjusting RTO — retransmissions still occur. > - Not capturing the full context: missing tcpdump capture, not recording ss output, or omitting ethtool stats. > - Forgetting to document the root cause: failure persists, but the fix is not documented for future use.

Final Notes

  systemctl status network.target
  systemctl status systemd-networkd
  systemctl status systemd-resolved
  #!/bin/bash
  # /opt/troubleshoot/network-health.sh

  echo "Network Health Report $(date)"
  echo "--------------------------------"
  ping -c 5 -i 0.5 -W 3 -s 1024 example.com
  traceroute -q 3 -m 30 -w 5 example.com
  ss -tuln
  ip route show
  ip addr show
  tcpdump -i any -n -s 0 -c 100 -w /tmp/health.pcap 'port 80'

> Sharp edges to remember: > - ping does not use UDP, only ICMP — so ping may succeed while UDP traffic fails. > - traceroute uses ICMP echo by default, but UDP traceroute is more accurate for application paths. > - tcpdump captures packets from the wire, but tshark can decode TLS, HTTP/2, and QUIC — use both. > - ss is faster than netstat, but netstat shows more verbose output — use ss for quick checks, netstat for deep analysis.

Network troubleshooting is not a sequence of commands — it is a process of gathering, interpreting, and refining evidence until the network speaks clearly.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.