Performance Tuning

Measuring and improving latency and throughput honestly: why the mean lies, why percentiles do not average, coordinated omission, the USE and RED framings, profiling, the standard bottleneck classes, and the caching and retry behaviour that turns a small problem into an outage.

Performance tuning addresses the process of identifying and resolving bottlenecks in systems to meet latency, throughput, or resource utilization requirements. This work typically begins when a system exhibits measurable degradation under expected load or when operational metrics exceed established thresholds.

Identify Performance Bottlenecks

Use top or htop to examine CPU usage and identify processes consuming excessive resources:

top -p 12345

Monitor memory consumption with free:

free -h

Check disk I/O using iostat:

iostat -x 1 5

For network performance, use ss to inspect active connections:

ss -tuln

Examine system logs for patterns of errors or warnings related to timeouts, deadlocks, or resource exhaustion.

Database Performance Optimization

Use SQL query execution plans to identify slow queries:

EXPLAIN ANALYZE SELECT * FROM table WHERE column = 'value';

Monitor database connection pool usage. For PostgreSQL, check:

SELECT count(*) FROM pg_stat_activity;

Set appropriate work_mem settings for sorting operations:

SET work_mem = '256MB';

Adjust shared_buffers parameter in postgresql.conf:

shared_buffers = 256MB

Application-Level Tuning

Review application logs for error patterns or timeouts. Look for repeated stack traces or specific error messages indicating resource contention.

Use strace to trace system calls made by an application:

strace -p 12345 -o trace.log

Analyze garbage collection behavior in Java applications with JVM flags:

-XX:+PrintGC -XX:+PrintGCDetails -Xloggc:gc.log

Container and Resource Management

Set CPU limits for containers using Kubernetes resource specifications:

resources:
  limits:
    cpu: "1"
    memory: 512Mi

Adjust container memory limits with Docker:

docker run --memory=512m myapp

Monitor container resource usage with docker stats:

docker stats

System-Level Configuration

Tune kernel parameters for better I/O performance:

echo 'vm.dirty_ratio = 15' >> /etc/sysctl.conf
sysctl -p

Adjust TCP buffer sizes:

echo 'net.core.rmem_max = 134217728' >> /etc/sysctl.conf

Memory and Garbage Collection Tuning

Set appropriate heap size for Java applications using JVM flags:

-Xms2g -Xmx4g

Configure garbage collection algorithms:

-XX:+UseG1GC -XX:MaxGCPauseMillis=200

Monitor memory usage with pmap:

pmap -x 12345

Network Performance Optimization

Use netstat to check for connection backlog or dropped packets:

netstat -i

Tune TCP settings for high-throughput scenarios:

echo 'net.ipv4.tcp_congestion_control = bbr' >> /etc/sysctl.conf

Cache and Buffer Tuning

Configure application-level caches with specific TTL settings. For instance, in Python using Redis:

redis_client.setex('key', 3600, 'value')

Adjust buffer sizes for file I/O operations. On Linux, tune readahead:

blockdev --setra 1024 /dev/sda

Monitoring and Profiling

Use perf to profile CPU usage in real time:

perf top -p 12345

Capture profiling data with perf record:

perf record -p 12345 -o profile.data

Analyze profiling output:

perf report -i profile.data

Load Testing and Simulation

Run load tests using standard tools like wrk:

wrk -t12 -c400 -d30s http://example.com

Simulate traffic patterns with ab (Apache Bench):

ab -n 10000 -c 100 http://example.com/

Process and Thread Optimization

Monitor thread count with:

ps -eLf | wc -l

Adjust process priority using nice:

nice -n -5 command

Use ulimit to set process resource limits:

ulimit -u 4096

Storage and I/O Optimization

Use fio to benchmark disk performance:

fio --name=test --direct=1 --rw=randread --bs=4k --size=1G --numjobs=4 --runtime=60

Monitor disk usage with df:

df -h

Resource Allocation and Scheduling

Adjust process scheduling priorities using chrt:

chrt -f 99 command

Set CPU affinity for specific processes:

taskset -c 0,1 command

Memory Fragmentation

Monitor memory fragmentation with:

cat /proc/zoneinfo

Identify fragmented memory areas using vmstat:

vmstat 1 5

Database Indexing and Query Optimization

Create indexes on frequently queried columns:

CREATE INDEX idx_column ON table(column);

Drop unused indexes to reduce write overhead:

DROP INDEX index_name;

Logging and Diagnostic Overhead

Reduce logging verbosity during performance-critical periods:

logger -p local0.info "Performance tuning in progress"

Use structured logging formats to minimize parsing overhead.

Configuration Validation

Verify configuration changes with:

systemctl daemon-reload
systemctl restart service

Check service status after changes:

systemctl status service

Performance Regression Testing

Automate performance measurements using shell scripts:

#!/bin/bash
time curl -s http://example.com/api/endpoint

Monitor response times over time with curl and jq:

curl -w "@curl-format.txt" -o /dev/null -s http://example.com

Common Failure Modes

Silent performance degradation occurs when memory leaks are not caught. The application may appear stable while gradually consuming more resources.

Incorrect CPU limits can cause processes to be throttled unexpectedly, leading to increased latency without clear error messages.

Misconfigured database settings such as max_connections can cause connection pool exhaustion with no immediate visible errors.

Network configuration issues like MTU mismatches can silently introduce packet fragmentation and increased latency.

Poorly tuned garbage collection settings can cause application pauses that are not obvious in standard monitoring.

Incorrect buffer sizes can lead to excessive context switches or system call overhead.

Failure to account for I/O scheduling differences between storage types can result in inconsistent performance under load.

Performance Tuning Checklist

Testing and Validation

Perform performance validation with tools like stress:

stress --cpu 4 --timeout 30s

Use stress-ng for more comprehensive system stress testing:

stress-ng --vm 4 --timeout 60s

Monitor service response times during tuning:

while true; do curl -w "%{time_total}\n" -o /dev/null -s http://example.com/; done

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.