Network Traffic Analysis

Analyze network traffic patterns to detect anomalies, optimize performance, and enhance security. Covers flow analysis, packet inspection, traffic classification, bandwidth planning, and the patterns that turn raw network data into actionable intelligence.

Network Traffic Analysis is the practice of capturing, inspecting, and interpreting network data to diagnose connectivity issues, optimize performance, and validate system behavior. It matters when a service fails to respond, when latency spikes mysteriously, when a client reports intermittent timeouts, or when a new deployment introduces subtle regressions across a distributed system. It is the engineer's microscope for the digital nervous system.

Understanding Traffic Flow

To understand how traffic flows through a system, engineers begin by capturing packets and analyzing their structure. The most common tool is tcpdump, which can filter, capture, and write traffic to disk with minimal overhead.

tcpdump -i eth0 -n -s 0 -w /tmp/traffic.pcap 'host 10.0.1.10 and port 8080'

This command captures all packets on interface eth0, resolves IP addresses numerically (-n), captures full packet payloads (-s 0), and writes them to a file using the pcap format. The filter expression selects traffic to and from 10.0.1.10 on port 8080.

For real-time inspection, engineers use tshark, the command-line version of Wireshark, which can parse and display protocol hierarchies:

tshark -r /tmp/traffic.pcap -T fields -e frame.time -e ip.src -e ip.dst -e tcp.srcport -e tcp.dstport -e http.request.method -e http.response.code

This extracts specific fields from a pcap file, producing a tabular output:

14:32:15.234567    10.0.1.5    10.0.1.10    54321       8080        GET         200
14:32:15.345678    10.0.1.10   10.0.1.5     8080        54321       POST        201

The key insight from this output is the timing and round-trip pattern: when a request arrives, how long until the response is sent, and whether the server processes multiple requests in parallel.

Diagnosing Latency and Delays

Latency manifests in two ways: *network delay* and *server processing delay*. Engineers distinguish them by measuring the time between packet arrivals.

To isolate network delay, use ping with precise timing and packet size:

ping -c 10 -i 0.5 -s 1024 -W 3 10.0.1.10

This sends 10 ICMP packets every 0.5 seconds, each of 1024 bytes, with a 3-second timeout per packet. The output includes:

A high jitter indicates variable network conditions, often due to congestion or routing changes. A sustained packet loss suggests link instability or buffer overflow.

For deeper analysis, engineers use iperf3 to measure throughput and latency under load:


iperf3 -s -p 5201 -i 1

# On the client
iperf3 -c 10.0.1.10 -p 5201 -t 60 -i 2 -b 10M -l 1024

This runs a 60-second test with 10 Mbps bandwidth, 1024-byte send buffers, and 2-second intervals. The output includes:

A mismatch between expected and actual throughput reveals bottlenecks: a 100 Mbps link with only 40 Mbps throughput indicates congestion, possibly due to TCP window size limits or slow receiver processing.

Identifying and Resolving Packet Loss and Retransmissions

Packet loss is often invisible until it causes retransmissions in TCP. Engineers use tcpdump with tcp.analysis.retransmission to detect retransmissions:

tcpdump -i eth0 -n -r /tmp/traffic.pcap -s 0 'tcp[tcpflags] & (tcp-ack|tcp-rst) == 0' -A

This filters for packets that are not ACKs or RSTs, highlighting data packets. The -A flag shows ASCII content, making it easy to spot lost segments.

To correlate retransmissions with latency, engineers use tshark with a custom display filter:

tshark -r /tmp/traffic.pcap -Y 'tcp.analysis.retransmission' -T fields -e frame.number -e tcp.seq -e tcp.len -e tcp.analysis.retransmission

This extracts frame numbers, sequence numbers, packet lengths, and retransmission flags. A retransmitted packet appears with tcp.analysis.retransmission set, and the sequence number is usually the same as a previously sent packet.

A common failure mode is delayed ACKs causing the receiver to acknowledge packets late, which leads to delayed feedback and inefficient use of bandwidth. Engineers verify this by examining TCP ACK timing:

tshark -r /tmp/traffic.pcap -Y 'tcp.flags.ack == 1' -T fields -e frame.time -e tcp.ack -e tcp.len

If ACKs arrive every 100 ms instead of every 20 ms, and the sender only sends 1000 bytes between ACKs, then the system is likely using a delayed ACK timer of 100 ms.

Debugging Connection Establishment and Handshake Issues

The TCP handshake is a frequent source of failure. Engineers use tcpdump to capture the full three-way handshake:

tcpdump -i eth0 -n -s 0 -w /tmp/handshake.pcap 'tcp[tcpflags] == 0x02'

This captures packets with the SYN flag set, which includes the initial SYN, SYN-ACK, and ACK from the client and server.

To analyze the handshake, engineers use tshark to extract the sequence number progression:

tshark -r /tmp/handshake.pcap -Y 'tcp.flags.syn == 1' -T fields -e frame.number -e tcp.seq -e tcp.window_size -e tcp.options.mss

A mismatch between the MSS (Maximum Segment Size) advertised in the SYN and SYN-ACK packets indicates a misconfigured MTU. If the client sends an MSS of 1460 but the server replies with 1400, and packets are being fragmented, then the MTU is likely 1500, and one or both endpoints are not negotiating MTU properly.

For TLS handshakes, engineers use openssl s_client with verbose output:

openssl s_client -connect 10.0.1.10:443 -servername app.example.com -debug -msg -tlsextdebug

This connects to a server and shows all TLS messages, including:

The -msg flag prints each TLS message with its payload, while -tlsextdebug shows details of extensions like SNI, ALPN, and supported versions. A missing ALPN extension in a server response indicates that the client sent a preferred protocol (e.g., HTTP/2) but the server did not advertise support.

Tracking Request-Response Patterns Across Services

In microservices, engineers trace a request from client to backend, often across multiple hops. They use curl to simulate real traffic and capture the full path:

curl -v -H 'X-Request-ID: 12345' -H 'Content-Type: application/json' \
     -d '{"query": "users"}' \
     http://api.example.com/v1/users

The -v flag enables verbose output, showing the full request and response headers, timing, and body. The X-Request-ID header allows correlation across services.

To validate that this request is logged and propagated through a chain of services, engineers use tcpdump on each service's network interface and correlate timestamps:

tcpdump -i eth0 -n -s 0 -w /tmp/api-traffic.pcap 'host api.example.com and port 8080'

They then extract request and response times using tshark:

tshark -r /tmp/api-traffic.pcap -Y 'http.request' -T fields -e http.request.method -e http.request.uri -e frame.time -e tcp.len -e http.response.code

A request that takes 500 ms from client to server, but the server logs a 300 ms processing time, implies that network delays (e.g., DNS resolution, TLS handshake, TCP setup) account for 200 ms.

Detecting and Troubleshooting HTTP/2 and HTTP/3 Behavior

When using HTTP/2 or HTTP/3, engineers verify multiplexing, flow control, and stream prioritization.

To capture HTTP/2 traffic, they use tcpdump with BPF filters for HTTP/2 frames:

tcpdump -i eth0 -n -s 0 -w /tmp/http2.pcap 'tcp port 443 and (tcp[((tcp[12:1] & 0xf0) >> 2)] == 0x10)'

This captures packets where the first byte of the TCP payload is 0x10 (HTTP/2 SETTINGS frame). They then use tshark to parse HTTP/2 frames:

tshark -r /tmp/http2.pcap -Y 'http2.frame.type == 1' -T fields -e frame.number -e http2.frame.stream_id -e http2.frame.length -e http2.headers.path

A stream ID that jumps from 1 to 3 to 5 indicates stream creation and interleaving. A frame.length of 1000 but only 100 bytes in the payload suggests flow control is active.

For HTTP/3 (QUIC), engineers capture UDP traffic and analyze QUIC frames:

tcpdump -i eth0 -n -s 0 -w /tmp/http3.pcap 'udp port 443'

They extract QUIC connection IDs and stream IDs:

tshark -r /tmp/http3.pcap -Y 'quic.type == 0x01' -T fields -e quic.connection_id -e quic.stream_id -e quic.version -e quic.packet_number

A missing quic.type == 0x01 (Initial packet) indicates that the client did not receive a server’s initial response. A jump in quic.packet_number without corresponding quic.type == 0x02 (Handshake) suggests that the client sent a handshake packet but the server did not acknowledge it.

Correlating Traffic with System Metrics

Traffic analysis is most powerful when combined with system-level metrics. Engineers use journalctl to correlate network events with application logs:

journalctl -u myapp.service --since "2023-10-05T14:00:00" --until "2023-10-05T14:05:00" -f

This monitors logs from the myapp.service unit, filtering for events between 14:00 and 14:05. They cross-reference these logs with timestamps from tcpdump outputs.

For example, a spike in myapp.service logs at 14:02:17 corresponds to a burst of TCP packets at 14:02:17.456, which is captured in a tcpdump trace. The engineer checks the packet length and sequence numbers to determine whether the application is sending large responses, buffering data, or experiencing backpressure.

They also use ps aux and top to monitor CPU and memory during traffic spikes:

ps aux | grep myapp
top -H -p $(pgrep myapp)

A high CPU with many threads suggests that the application is handling concurrent requests. A growing memory usage indicates that the application is caching data or not releasing buffers properly.

Common Pitfalls and Silent Failures

The most common failure is missing TCP ACKs. A server sends a response, but the client never acknowledges it. This causes the sender to retransmit the same data, leading to unnecessary bandwidth and increased latency. Engineers detect this by examining tcp.analysis.ack_rtt in tshark:

tshark -r /tmp/traffic.pcap -Y 'tcp.analysis.ack_rtt' -T fields -e frame.time -e tcp.ack -e tcp.analysis.ack_rtt

If the ACK round-trip time is much longer than expected, it suggests that the client is processing responses slowly or that the network is congested.

Another silent failure is TCP window scaling. A server sends large amounts of data, but the client’s receive window does not increase, causing the sender to stall. Engineers check the TCP window size:

tshark -r /tmp/traffic.pcap -Y 'tcp.window_size' -T fields -e tcp.window_size -e tcp.window_size_scale_factor

A window size of 65535 but window_size_scale_factor of 2 indicates that the sender can send up to 262140 bytes before needing an ACK. If the client’s window remains at 65535, the server is underutilized.

Finally, engineers often confuse TCP delayed ACKs with TCP quick ACKs. Delayed ACKs are triggered by a timer (e.g., 200 ms) or a threshold (e.g., two packets). Quick ACKs are sent immediately upon receiving a packet. To detect this, they use tshark with tcp.analysis.ack_rtt and compare the RTT of the first and second ACKs in a sequence.

A second ACK with an RTT of 50 ms suggests a quick ACK, while a 200 ms RTT indicates a delayed ACK. If both are used, the system is efficient; if only delayed ACKs are used, the system is underperforming.

Summary

Network Traffic Analysis is the foundation of diagnosing and optimizing network behavior. Engineers capture traffic with tcpdump, inspect it with tshark, and correlate it with application logs and system metrics. They identify latency, packet loss, retransmissions, handshake issues, and protocol-specific behaviors in HTTP/2 and HTTP/3. They detect subtle failures—missing ACKs, window scaling, delayed ACKs—and validate their findings with precise, real-world commands and observations. The result is a deep, actionable understanding of how data moves through the system.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.