Network Monitoring Tools

Production engineering guide for network monitoring tools covering patterns, implementation strategies, and operational best practices.

Network monitoring tools are the eyes and ears of a network infrastructure. They detect, diagnose, and alert on failures in real time—link outages, latency spikes, packet loss, routing instability, and service degradation. These tools matter when a service is down, when a latency threshold is breached, or when a new deployment causes cascading network issues. They are essential during incident response, capacity planning, and proactive maintenance.

Monitoring Network Performance and Latency

To measure end-to-end network performance, use ICMP-based probing tools that send regular pings and collect metrics such as round-trip time, packet loss, jitter, and path MTU. The most common approach is to run ping with precise timing and statistics.

ping -c 100 -i 0.5 -W 2 -s 1472 -M do example.com

This command reports per-packet RTT, loss, and jitter. The output includes a summary line:

--- example.com ping statistics ---
100 packets transmitted, 98 received, 2% packet loss, time 49737ms
rtt min/avg/max/mdev = 15.321/18.643/25.128/1.875 ms

For continuous monitoring, combine ping with mtr (My Traceroute) to track path changes and packet loss per hop.

mtr -c 1000 -i 0.5 -r -w example.com

The output shows cumulative packet loss, average RTT, and real-time updates. Use mtr with --report mode for structured output that can be parsed by monitoring systems.

To measure HTTP-level latency and availability, use curl with timing options and check response status.

curl -s -w 'time_total:%{time_total}\ntime_namelookup:%{time_namelookup}\ntime_connect:%{time_connect}\ntime_appconnect:%{time_appconnect}\ntime_pretransfer:%{time_pretransfer}\ntime_redirect:%{time_redirect}\n' \
     -H 'Accept: application/json' \
     --max-time 5 \
     https://api.example.com/v1/users \
     -o /dev/null

The time_total field includes DNS resolution, TCP handshake, TLS negotiation, request transmission, and response reception.

Detecting and Diagnosing Packet Loss and Jitter

Packet loss and jitter are critical in real-time applications like VoIP, video conferencing, and live streaming. Use ping with large payloads and long durations to detect intermittent loss.

ping -c 10000 -i 1 -s 1500 -l -W 3 -n example.com

Use tcpdump to capture and analyze packet loss at the link layer.

tcpdump -i eth0 -s 0 -w /var/log/packets-loss.pcap -C 1024 -W 10 -G 300 -B 4096 -Z root

This setup captures 5-minute intervals of network traffic, ideal for diagnosing bursty loss or jitter. Analyze with tshark (command-line version of Wireshark):

tshark -r /var/log/packets-loss.pcap -T fields -e frame.time_epoch -e ip.src -e ip.dst -e icmp.type -e icmp.code -e icmp.id -e icmp.seq -e frame.len

Use tshark in a pipeline to detect jitter and loss:

tshark -r /var/log/packets-loss.pcap \
  -T fields \
  -e frame.time_epoch \
  -e icmp.id \
  -e icmp.seq \
  -e frame.len \
  -e icmp.time \
  | awk '
    BEGIN { last_time = 0; lost = 0; total = 0; jitter_sum = 0 }
    {
      time = $1; id = $2; seq = $3; len = $4; rtt = $5
      if (id in seen && seq == seen[id] + 1) {
        seen[id] = seq
        total++
        jitter = (time - last_time) - (rtt - 1000000)
        jitter_sum += jitter
      } else {
        lost++
      }
      last_time = time
    }
    END {
      printf "Packets total: %d, lost: %d, loss rate: %.2f%%\n", total, lost, (lost / total) * 100
      printf "Jitter average: %.2f microseconds\n", jitter_sum / total
    }'

This script computes jitter as the difference between actual inter-packet arrival time and expected RTT.

End-to-End Service Monitoring with Synthetic Probes

Synthetic monitoring simulates real user traffic to test service availability and performance. Use curl scripts to emulate user workflows, then schedule them with systemd timers or cron.

Example: Check a login flow with curl.

#!/bin/bash


set -euo pipefail

URL="https://auth.example.com"
TOKEN_FILE="/tmp/auth-token.json"

# Step 1: Login
login_response=$(curl -s -w '%{http_code}' \
  -H 'Content-Type: application/json' \
  -d '{"username":"testuser","password":"secret"}' \
  -X POST \
  "$URL/api/login" \
  -o /dev/null)

HTTP_CODE=$(echo "$login_response" | tail -n1)
RESPONSE_BODY=$(echo "$login_response" | head -n-1)

# Step 2: Extract token from response
ACCESS_TOKEN=$(echo "$RESPONSE_BODY" | jq -r '.access_token')

# Step 3: Use token to fetch user profile
profile_response=$(curl -s -w '%{http_code}' \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -X GET \
  "$URL/api/profile" \
  -o /dev/null)

PROFILE_CODE=$(echo "$profile_response" | tail -n1)
PROFILE_BODY=$(echo "$profile_response" | head -n-1)

# Step 4: Log results
timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
echo "$timestamp | HTTP_LOGIN=$HTTP_CODE | HTTP_PROFILE=$PROFILE_CODE | PROFILE_DATA=$PROFILE_BODY" >> /var/log/synthetic-login.log

# Exit with non-zero if any step failed
if [ "$HTTP_CODE" -ne 200 ] || [ "$PROFILE_CODE" -ne 200 ]; then
  exit 1
fi

Schedule this script with systemd:

# /etc/systemd/system/synthetic-login.service
[Unit]
Description=Synthetic Login Monitoring
After=network.target

[Service]
Type=oneshot
ExecStart=/opt/monitoring/synthetic-login.sh
StandardOutput=journal
StandardError=journal
User=monitoring
Group=monitoring

[Install]
WantedBy=multi-user.target
# /etc/systemd/system/synthetic-login.timer
[Unit]
Description=Run synthetic login every 5 minutes
Requires=synthetic-login.service

[Timer]
OnCalendar=*-*-* *:*:00/5
Persistent=true

[Install]
WantedBy=timers.target

Enable and start:

systemctl enable synthetic-login.timer
systemctl start synthetic-login.timer

This setup runs a synthetic login every 5 minutes and logs results. The service fails if either the login or profile fetch returns a non-200 status.

Aggregating and Visualizing Network Metrics

Collect metrics across multiple hosts using collectd, telegraf, or a custom script. Each tool writes metrics in a standardized format—typically JSON or InfluxDB line protocol.

Example: A collectd configuration to monitor network interface statistics.

<!-- /etc/collectd/collectd.conf -->
<Plugin network>
  Listen "0.0.0.0"
  Port "25826"
  Server "monitoring.example.com" "25826"
  PluginDir "/usr/lib/collectd"
</Plugin>

<Plugin interface>
  Interface "eth0"
  Interface "eth1"
  Values "rx_bytes", "tx_bytes", "rx_packets", "tx_packets", "rx_errors", "tx_errors"
  ReportByDevice true
  ReportByType true
</Plugin>

<Plugin csv>
  PluginDir "/usr/lib/collectd"
  DataDir "/var/lib/collectd/csv"
  StoreRates true
  FlushInterval 60
  FlushInterval 120
</Plugin>

On the monitoring server, use influxdb to store and query metrics.

-- Query: average TX bytes per second on eth0 over the last 5 minutes
SELECT mean("tx_bytes") AS "avg_tx_bytes"
FROM "collectd"."autogen"."interface_tx_bytes"
WHERE "interface" = 'eth0'
  AND time > now() - 5m
GROUP BY time(1m)

Use grafana to visualize these metrics with a dashboard that shows:

Grafana can pull data from influxdb, prometheus, postgres, or elasticsearch using standard HTTP APIs.

Alerting on Network Events

Set up alerting rules based on thresholds, anomalies, or time-series patterns. Use alertmanager (Prometheus ecosystem) or grafana to trigger notifications.

Example: Alert when average RTT exceeds 50ms for more than 3 consecutive minutes.

# /etc/alertmanager/alerting.yml
route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'email'

receivers:
  - name: 'email'
    email_configs:
      - to: '[email protected]'
        from: '[email protected]'
        smarthost: 'smtp.company.com:587'
        auth_username: 'alertmanager'
        auth_password: 'secret'
        auth_identity: '[email protected]'
        send_resolved: true

inhibit_rules:
  - equal: ['alertname', 'cluster']
    source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['cluster']

Create a rule in Prometheus:

# /etc/prometheus/rules/monitoring.rules
groups:
  - name: network-latency
    rules:
      - alert: HighLatency
        expr: |
          avg by (instance) (
            rate(icmp_round_trip_time_seconds_sum[5m])
            / rate(icmp_round_trip_time_seconds_count[5m])
          ) > 0.05
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "High RTT on {{ $labels.instance }}"
          description: |
            Average RTT on {{ $labels.instance }} is above 50ms.
            Current value: {{ $value }} seconds.
            Last 5 minutes.

This rule triggers an alert when average RTT exceeds 50ms for 3 minutes. Alertmanager sends email and updates Grafana dashboards.

Debugging Network Topology and Routing Instability

Use traceroute with multiple probes and path tracing to detect routing changes.

traceroute -m 30 -q 3 -w 3 -n example.com

The output shows the path and RTT for each probe. Use tracepath for a lightweight alternative.

tracepath -n -m 30 -q 3 example.com

For BGP-level routing instability, use bgpmon or bird to track route changes and compute convergence times.

bgpmon -c /etc/bgpmon.conf -l /var/log/bgp-changes.log -r /var/log/bgp-stats.json

The tool tracks prefix announcements, withdrawals, and path changes, reporting convergence time and flapping patterns.

Use ip route show and ip route get to verify routing decisions.

ip route get 192.0.2.1

Output:

192.0.2.1 via 10.1.1.1 dev eth0 src 10.1.1.100
    cache

This indicates the route is learned via 10.1.1.1 on eth0, with source IP 10.1.1.100. Use ip route show table to inspect multiple routing tables.

Silent Failures and Sharp Edges

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.