Network Monitoring and Troubleshooting

Monitor and troubleshoot network issues in production environments. Covers monitoring tools, common failure patterns, packet analysis, latency diagnosis, and DNS troubleshooting.

Network monitoring and troubleshooting is the practice of observing, diagnosing, and resolving network issues as they occur, from latency spikes to complete outages. It matters when systems are live, users are depending on services, and a single packet loss can degrade experience or halt operations. This page covers the essential tasks engineers perform when networks behave unexpectedly, focusing on real tools, real commands, and real failure modes.

Detecting Network Outages

A network outage is not just a down service; it is a failure in reachability, connectivity, or performance across layers. The first step is to confirm that a service is unreachable and to quantify the failure.

Use ping with consistent interval and packet size to test reachability and round-trip time.

ping -c 10 -i 0.5 -s 1472 -W 5 example.com

A failed ping shows packet loss, high jitter, or timeout. When a host is unreachable, use traceroute to identify where the failure occurs.

traceroute -m 30 -q 3 -w 3 example.com

Observe:

For automated monitoring, use curl in a script to verify HTTP services:

curl -s -f -m 10 -o /dev/null -w "%{http_code} %{time_total}s\n" http://api.example.com/health

The output 200 0.345s indicates success; 503 10.012s suggests a timeout or upstream failure.

Diagnosing Latency and Jitter

Latency is not just a number—it is a symptom of many underlying problems. To isolate the cause, gather metrics across layers.

Use tcpdump to capture traffic and analyze packet timing:

sudo tcpdump -i eth0 -n -s 0 -w /var/log/latency.pcap host example.com and port 80

To analyze the capture, use tshark, the command-line version of Wireshark:

tshark -r /var/log/latency.pcap -T fields -e frame.time_delta -e ip.src -e ip.dst -e tcp.window_size -e tcp.analysis.rtt

Look for:

In production, latency is often caused by slow DNS, which can be confirmed with dig:

dig +time=5 +retry=3 +short @8.8.8.8 example.com

A response like 192.0.2.1 192.0.2.2 with a 3-second delay indicates DNS is the bottleneck.

For end-to-end latency, use mtr (My Traceroute), which combines ping and traceroute continuously:

mtr -c 100 -i 1 -w example.com

The output shows:

A spike in jitter on a single hop suggests variable bandwidth, bufferbloat, or queuing delay.

Troubleshooting Packet Loss and Corruption

Packet loss is often silent: a few packets go missing, but the application continues. To detect and diagnose loss, use tools that expose the full packet stream.

Run ping with large packets and high frequency:

ping -c 1000 -i 0.1 -s 4096 -W 3 -a example.com

A missing packet shows up as a gap in the audio or timing in the output.

Use iperf3 to test sustained throughput and detect loss during transfer:


iperf3 -s -p 5201 -D

# On client
iperf3 -c 192.0.2.1 -t 60 -i 1 -P 4 -b 100M -l 131072

Examine the output:

If throughput is below target, check for:

Use tcpdump to extract lost packets:

tcpdump -r /var/log/iperf3.pcap -n -A -s 0 'tcp port 5201'

Look for:

A packet loss pattern of retransmissions with increasing sequence numbers indicates a lossy link or high bufferbloat.

DNS is often the first point of failure in network troubleshooting. It is not enough to know that a name resolves; engineers must verify the path from resolver to authoritative server.

Use dig to query specific records and compare results across resolvers:

dig +trace +all example.com

The output shows:

Check for:

For recursive resolver analysis, use dig with specific resolver:

dig @1.1.1.1 example.com

Compare responses:

Use nslookup for quick checks:

nslookup -type=AAAA example.com 2001:4860:4860::8888

When DNS fails, verify that the resolver is reachable:

ping -c 5 -i 1 1.1.1.1

Use dig to test DNS over UDP and TCP:

dig +tcp +short example.com
dig +udp +short example.com

If a large response is truncated, use +tcp to ensure full delivery.

Investigating TCP and Transport Layer Problems

TCP is the backbone of modern network applications, but it is often misunderstood. Troubleshooting requires examining connection state, flow control, and retransmissions.

Use ss (socket statistics) to inspect active connections:

ss -tuln 'sport = :80'

Look for:

A high rmem or wmem indicates buffer pressure.

Use netstat to generate a snapshot of network state:

netstat -s -i -c 10

Examine:

A high number of duplicate ACKs indicates that the receiver is receiving out-of-order packets.

For deep TCP analysis, use tcpdump to capture connection setup:

tcpdump -i eth0 -n -s 0 -w /var/log/tcphandshake.pcap 'tcp[tcpflags] & (tcp-syn|tcp-ack) = tcp-syn'

Analyze the capture:

Use tc (traffic control) to simulate network conditions and reproduce issues:

tc qdisc add dev eth0 root handle 1: htb default 10
tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit
tc qdisc add dev eth0 parent 1:1 handle 10: netem delay 100ms loss 10% duplicate 5% reorder 10%

Apply this profile to a test server and run iperf3 to observe performance under stress.

Monitoring Network Health Continuously

Network monitoring is not a one-off task; it is an ongoing process. Engineers use a combination of tools to detect issues before users notice them.

Set up a systemd service to run periodic checks:

[Unit]
Description=Network Health Check
After=network.target

[Service]
Type=oneshot
ExecStart=/opt/monitoring/check-network.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

The script /opt/monitoring/check-network.sh contains:

#!/bin/bash

# Check ping
ping -c 5 -i 0.5 -s 1472 -W 5 example.com > /var/log/ping-check.log

# Check HTTP service
curl -s -f -m 10 -o /dev/null -w "%{http_code}" http://api.example.com/health >> /var/log/http-check.log

# Check DNS
dig +short example.com >> /var/log/dns-check.log

# Send alert on failure
if ! grep -q "200" /var/log/http-check.log; then
    echo "HTTP health check failed" | mail -s "Network Alert" [email protected]
fi

Schedule the service with systemctl:

sudo systemctl enable check-network.service
sudo systemctl start check-network.service

Use journalctl to review logs:

journalctl -u check-network.service -f

When an alert triggers, engineers can:

Each tool reveals a layer of insight, building a complete picture of network health.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.