Network Performance Tuning

Production engineering guide for network performance tuning covering patterns, implementation strategies, and operational best practices.

Network performance tuning is the process of identifying, diagnosing, and optimising network latency, throughput, and reliability across systems and services. It matters most during service ramp-up, after infrastructure changes, under high load, and when users report slowness with no clear application root cause.

Optimising End-to-End Latency

End-to-end latency is the time from when a client sends a request until it receives a complete response. Tuning begins with measuring and reducing each segment: client, network, server, and application.

Use curl with the --write-out option to extract detailed timing metrics:

curl -v --write-out '
  time_namelookup: %{time_namelookup}s
  time_connect: %{time_connect}s
  time_appconnect: %{time_appconnect}s
  time_pretransfer: %{time_pretransfer}s
  time_starttransfer: %{time_starttransfer}s
  time_total: %{time_total}s
  speed_download: %{speed_download}B/s
  speed_upload: %{speed_upload}B/s
' https://api.example.com/v1/users

This produces:

time_namelookup: 0.123s
time_connect: 0.211s
time_appconnect: 0.098s
time_pretransfer: 0.443s
time_starttransfer: 0.612s
time_total: 1.054s
speed_download: 102400B/s
speed_upload: 51200B/s

The time_namelookup measures DNS resolution time; if it exceeds 100ms, check DNS configuration. The time_connect includes TCP handshake; values above 200ms indicate network congestion or slow routing. time_appconnect covers TLS handshake; if it exceeds 100ms, consider optimizing TLS configuration.

To reduce TLS overhead, configure nginx with:

ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;

Enable HTTP/2 in nginx with:

listen 443 http2 ssl;

Clients must support HTTP/2; otherwise, time_starttransfer remains high due to HTTP/1.1’s serial request-response model.

Reducing Packet Loss and Retransmissions

Packet loss causes retransmissions and increased latency. Use ping with fixed payload and interval to measure loss:

ping -c 100 -i 0.1 -s 1472 -M do -W 2 -t 255 192.168.1.1

This sends 100 ICMP packets of 1472 bytes (1500-byte MTU), with 100ms intervals, DF bit set, 2s timeout, and TTL 255. The output includes:

--- 192.168.1.1 ping statistics ---
100 packets transmitted, 94 received, 6% packet loss, time 9910ms
rtt min/avg/max/mdev = 1.234/1.876/4.567/0.345 ms

A packet loss rate above 1% suggests network congestion. Use tcpdump to identify retransmissions:

tcpdump -i eth0 -n -s 0 -v -B 1000 -w capture.pcap 'tcp and (tcp.flags & 0x10) != 0'

This captures packets with the ACK flag set, which helps identify ACK packets sent in response to retransmissions. Examine the tcp.analysis.retransmission field in Wireshark or with tshark:

tshark -r capture.pcap -T fields -e frame.number -e tcp.analysis.retransmission -e tcp.analysis.duplicate_ack

A retransmission occurs when a packet is sent and received, but the receiver sends an ACK for a different sequence number. Duplicated ACKs appear when multiple ACKs are sent for the same sequence number.

To reduce retransmissions, tune TCP parameters on Linux using sysctl:

net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 16384 134217728
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fack = 1

The tcp_rmem and tcp_wmem arrays set minimum, default, and maximum buffer sizes for receive and send windows. Enable window scaling (tcp_window_scaling) to support large window sizes (beyond 65,535 bytes). Timestamps (tcp_timestamps) enable RTT estimation and reduce the chance of tail losses.

Improving Throughput with Window Scaling and Flow Control

Throughput depends on the TCP window size and sender’s ability to keep the pipe full. Measure effective bandwidth using iperf3:

iperf3 -c 192.168.1.1 -t 30 -i 1 -P 4 -l 128k -b 100M

This runs a 30-second test with 4 parallel streams, 128KB buffer size, and a 100 Mbps bandwidth limit. The output includes:

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  5] 0.00-10.00 sec  1.15 GBytes  988 Mbits/sec  1234  4096 KBytes
[  6] 0.00-10.00 sec  1.14 GBytes  983 Mbits/sec  1235  4096 KBytes
[  7] 0.00-10.00 sec  1.16 GBytes  993 Mbits/sec  1236  4096 KBytes
[  8] 0.00-10.00 sec  1.15 GBytes  985 Mbits/sec  1237  4096 KBytes

If the measured bandwidth is less than the link capacity, increase the TCP window size. The cwnd (congestion window) should stabilize at or near the target.

Adjust the TCP receive buffer size in nginx:

http {
    tcp_nopush on;
    tcp_nodelay on;
    sendfile on;
    aio threads;
    directio 512k;
    keepalive_timeout 65;
    client_body_buffer_size 128k;
    client_max_body_size 10M;
}

Use ss to inspect current TCP connections and their window sizes:

ss -i -t -a -n

Output includes:

State      Recv-Q Send-Q Local Address:Port               Peer Address:Port
ESTAB      0      0      192.168.1.10:80                  192.168.1.1:54321
    local tcp rto:200 rtt:15.3 cwnd:16 ssthresh:16
    remote tcp rto:250 rtt:20.4 cwnd:16 ssthresh:16

The cwnd field shows the current congestion window. A small value (e.g., 8) indicates the sender is not sending enough data to fill the pipe. A cwnd that grows slowly suggests a slow-start phase or congestion.

Tuning for High-Frequency, Low-Latency Workloads

For services that require frequent, low-latency interactions—such as real-time dashboards, trading systems, or gaming—reduce the number of round trips.

Use HTTP/2 with multiplexing and server push. Configure nginx to push resources:

location / {
    http2_push /css/main.css;
    http2_push /js/main.js;
    http2_push /images/logo.png;
}

Clients receive the main HTML document and immediately get CSS, JS, and image files without additional requests.

To further reduce latency, enable TCP Fast Open (TFO) on Linux:

net.ipv4.tcp_fastopen = 3

This enables both client and server TFO. Use tcpdump to verify TFO packets:

tcpdump -i eth0 -n -s 0 -v -A 'tcp[tcpflags] & (tcp-fin|tcp-syn) != 0 and tcp[tcpflags] & tcp-ack != 0'

TFO packets include a TCP Option: Fast Open (0x02) with a cookie and data fields. The first packet of a connection carries data, reducing the latency of the first request.

For services with bursty traffic, tune nginx to handle many short-lived connections:

events {
    worker_connections 8192;
    use epoll;
    multi_accept on;
    accept_mutex on;
    accept_mutex_delay 100ms;
}

http {
    keepalive_timeout 15 15;
    keepalive_requests 1000;
    client_body_timeout 30s;
    client_header_timeout 30s;
}

Set multi_accept on so that each worker accepts multiple connections in a single event loop iteration. accept_mutex on prevents thundering herd problems, and accept_mutex_delay allows workers to batch accept calls.

Diagnosing and Resolving Network Congestion

Congestion arises when the network cannot handle the incoming data rate. Use tc (traffic control) to simulate and measure congestion.

Define a rate-limited queue on a network interface:

tc qdisc add dev eth0 root handle 1: htb default 10
tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit ceil 100mbit
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip dst 192.168.1.10 flowid 1:1

This creates a hierarchical token bucket (HTB) queue with a 100 Mbps rate limit, applying it to traffic destined for 192.168.1.10.

Monitor congestion with nstat:

nstat -a -c "TCP.*"

This shows all TCP statistics, including:

High TcpFastRetrans values indicate frequent fast retransmissions. A large TcpInRecovery value suggests prolonged congestion recovery.

To resolve congestion, increase the tcp_adv_win_scale and tcp_rfc1323 settings:

net.ipv4.tcp_adv_win_scale = 2
net.ipv4.tcp_rfc1323 = 1

This enables RFC 1323 window scaling and allows the receiver to advertise a window size that is scaled by a factor of 2^tcp_adv_win_scale. Combined with tcp_window_scaling, this allows window sizes up to 1 GB.

Handling Jitter and Asynchronous Workloads

Jitter—variation in packet inter-arrival times—causes uneven data flow and affects real-time applications. Measure jitter using ping with detailed timing:

ping -c 100 -i 0.1 -s 1472 -M do -W 2 -t 255 -Q 100 192.168.1.1

The -Q 100 flag sets the priority to 100, reducing jitter in the network stack. Use iperf3 with jitter tracking:

iperf3 -c 192.168.1.1 -t 30 -i 1 -P 4 -l 128k -b 100M --jitter

Output includes:

Jitter: 1.234ms

High jitter (above 5ms) indicates network instability. To reduce jitter, enable QoS (Quality of Service) on the network switch using DSCP (Differentiated Services Code Point):

tc qdisc add dev eth0 root handle 1: htb default 10
tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit ceil 100mbit prio 0
tc class add dev eth0 parent 1: classid 1:2 htb rate 20mbit ceil 50mbit prio 1
tc class add dev eth0 parent 1: classid 1:3 htb rate 5mbit ceil 10mbit prio 2

tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip dscp 0x2e flowid 1:1  # VoIP
tc filter add dev eth0 protocol ip parent 1:0 prio 2 u32 match ip dscp 0x30 flowid 1:2  # Video
tc filter add dev eth0 protocol ip parent 1:0 prio 3 u32 match ip dscp 0x00 flowid 1:3  # Data

This prioritizes DSCP values: 0x2e (VoIP), 0x30 (Video), and 0x00 (Data). Packets with higher DSCP are served from the higher-priority queues.

Use tcpdump to verify DSCP values:

tcpdump -i eth0 -n -s 0 -v -A 'ip dscp 0x2e'

This captures only VoIP packets, helping verify that QoS is active.

When All Else Fails: The Role of TCP Congestion Control

When throughput and latency remain suboptimal, consider the TCP congestion control algorithm. Linux supports multiple algorithms: reno, cubic, bbr, hybla, westwood.

Check current algorithm:

cat /proc/sys/net/ipv4/tcp_congestion_control

Output: cubic.

Switch to bbr (Bottleneck Bandwidth and Round-trip propagation time):

echo bbr > /proc/sys/net/ipv4/tcp_congestion_control

Use ss to monitor BBR-specific metrics:

ss -i -t -a -n | grep -A 5 -B 5 'bbr'

Output includes:

    local tcp bbr
        bbr_bw: 12345678 bbr_min_rtt: 12 ms bbr_pacing_gain: 2.0 bbr_cwnd_gain: 2.0

bbr_bw shows the estimated bottleneck bandwidth. If it increases slowly, BBR is probing for new bandwidth. A high bbr_min_rtt indicates RTT fluctuations.

To ensure BBR is active across restarts, set it in sysctl:

net.ipv4.tcp_congestion_control = bbr

Bottleneck bandwidth estimation and pacing gain adjustments are automatic, but fine-tuning bbr_pacing_gain and bbr_cwnd_gain can improve performance in specific environments.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.