Load Testing: Knowing Your Breaking Point Before Production Does

Load testing is the practice of simulating real-world traffic patterns against a system under test to identify performance bottlenecks, resource limits, and failure modes before deployment. It matters when you need to understand how your application behaves under stress, particularly during traffic spikes or when scaling to new user volumes.

Testing Under Simulated Load

To simulate load on an HTTP API endpoint using standard tools:

# Use hey (https://github.com/rakyll/hey) for simple load testing
hey -n 1000 -c 100 https://api.example.com/users

# Use wrk (https://github.com/wg/wrk) for more advanced scenarios
wrk -t12 -c400 -d30s https://api.example.com/users

When using hey, the output includes:

Summary:
  Total:	30.0026 secs
  Requests/sec:	333.31
  Latency:	150.23ms
  90th percentile:	200.00ms

The -n flag specifies total requests, -c sets concurrent connections, and -d for duration in wrk.

If the system under test returns a 503 Service Unavailable, it may indicate resource exhaustion or request queuing.

Identifying Resource Constraints

Use kubectl top to monitor pod resource usage during tests:

kubectl top pods -n production

A pod may show:

NAME                     CPU(cores)   MEMORY(bytes)
web-7b5c9f8d4-xyz12      100m         256Mi
worker-6f7b8c9d2-abc34   500m         1Gi

If CPU usage spikes to 150% of limit, or memory exceeds the set limit, you've found a resource constraint.

Testing for Latency Degradation

To test latency under load with specific duration and concurrency:

# Use wrk with custom script to measure latency
wrk -t10 -c100 -d60s --latency https://api.example.com/users

If you see:

Latency Distribution (msec)
   50%    20.00
   75%    40.00
   90%    80.00
   95%   120.00
   99%   200.00

This indicates acceptable latency at 99th percentile under sustained load.

Testing API Response Times

Use curl with time tracking for granular response analysis:

curl -w "@curl-format.txt" -o /dev/null -s https://api.example.com/users

Where curl-format.txt contains:

time_namelookup:  %{time_namelookup}\n
time_connect:     %{time_connect}\n
time_appconnect:  %{time_appconnect}\n
time_pretransfer: %{time_pretransfer}\n
time_redirect:    %{time_redirect}\n
time_starttransfer: %{time_starttransfer}\n
time_total:       %{time_total}\n

A high time_total may indicate database or service-level bottlenecks.

Testing for Concurrent Connection Limits

Test with many concurrent connections to detect connection pool exhaustion:

# Use ab (Apache Bench) to simulate high concurrent load
ab -n 1000 -c 500 https://api.example.com/users

If ab returns:

Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      1234567 bytes
HTML transferred:       1234567 bytes
Requests per second:    20.00 [#/sec] (mean)
Time per request:       5000.00 [ms] (mean)

A high Time per request may indicate connection handling issues or upstream resource limits.

Testing Database Query Performance

Use a tool like pgbench to test PostgreSQL under load:

# Initialize database for testing
pgbench -i -s 10 mydb

# Run benchmark with concurrent connections
pgbench -c 50 -j 10 -T 60 mydb

If the database returns:

transaction type: <builtin: TPC-B (sort of)>
scaling factor: 10
query mode: simple
number of clients: 50
number of threads: 10
duration: 60 s
number of transactions actually processed: 1234
latency average: 200.00 ms

A latency increase or transaction count drop may indicate database resource contention.

Testing for Memory Leaks

Monitor memory consumption over time using system tools:

# Use top or htop to watch RSS
top -p $(pgrep myapp)

If the process RSS grows steadily during a long-running test, it may indicate a memory leak.

Testing HTTP Request Handling Limits

Use ab or wrk to test request body size limits:

# Send a large payload to check for limits
curl -X POST -d @large-payload.json https://api.example.com/upload

If the server returns:

{"error": "Request Entity Too Large", "code": 413}

You've found an HTTP request body limit.

Testing Timeout Behavior

Set timeouts explicitly to understand how your system reacts to slow requests:

# Use curl with timeout
curl --max-time 10 https://api.example.com/slow-endpoint

If the response is:

curl: (28) Operation timed out after 10001 milliseconds with 0 bytes received

You've confirmed a request timeout behavior.

Testing for Circuit Breaker Effects

Simulate high load to trigger circuit breaker patterns:

# Use hey to send high volume
hey -n 5000 -c 200 https://api.example.com/endpoint

If you see:

Status code distribution:
[200] 4900 responses
[503] 100 responses

You've identified a circuit breaker or service overload.

Testing for Queue Backlogs

Monitor queue depth if your system uses message queues:

# Check RabbitMQ queue length
rabbitmqctl list_queues name messages consumers

A queue with increasing messages and low consumers may indicate processing bottlenecks.

Testing Response Size Limits

Test maximum response size limits:

# Send request expecting large JSON response
curl -H "Accept: application/json" https://api.example.com/large-data

If you get:

{"error": "Response Too Large", "code": 413}

You've identified an HTTP response body size constraint.

Testing for Rate Limiting

Simulate burst traffic to observe rate limiting behavior:

# Use hey to send bursts
hey -n 1000 -c 500 https://api.example.com/endpoint

If you see:

Status code distribution:
[429] 100 responses
[200] 900 responses

You've confirmed rate limiting in place.

Testing Failure Modes

To simulate system failures during load:

# Test graceful degradation
hey -n 1000 -c 500 https://api.example.com/failing-endpoint

If the system returns:

{"error": "Service Unavailable", "code": 503}

You've identified failure modes under high load.

Testing Throughput Limits

Measure maximum throughput with increasing load:

# Run multiple tests with different concurrent connections
for c in 10 50 100 200; do
  echo "Testing with $c connections"
  hey -n 1000 -c $c https://api.example.com/endpoint
done

If throughput plateaus or drops, you've found a capacity limit.

Testing for Resource Exhaustion

Monitor system resources during load testing:

# Monitor disk I/O with iostat
iostat -x 1 5

If await values increase and util approaches 100%, you're hitting I/O limits.

Testing for Network Bottlenecks

Use tools to measure network performance under load:

# Test with iperf3 to measure bandwidth
iperf3 -c server-ip -t 60

If bandwidth is limited or latency increases, you've found a network constraint.

Testing for CPU Saturation

Monitor CPU usage during load:

# Use top or htop
top -p $(pgrep myapp)

If CPU utilization remains at 100% and response times increase, you're hitting CPU limits.

Testing for Memory Constraints

Monitor memory allocation:

# Use free command
free -m

If available memory drops to near zero during tests, the system is memory-constrained.

Testing for Thread Pool Exhaustion

If your application uses thread pools, simulate high load:

# Run with many concurrent requests
hey -n 1000 -c 200 https://api.example.com/threaded-endpoint

If you see increased latency or error codes, you've likely exhausted the thread pool.

Testing for Deadlock Scenarios

Run long-running tests to detect deadlocks:

# Run a long test with sustained load
wrk -t10 -c200 -d300s https://api.example.com/endpoint

If your application becomes unresponsive or hangs, you've found a potential deadlock.

Testing for Garbage Collection Impact

Monitor GC behavior during load:

# Use jstat for Java applications
jstat -gc <pid> 1s

If FGC or OGC counts increase significantly under load, garbage collection is affecting performance.

Testing for Database Connection Pool Limits

Check database connection usage:

# Use psql to check active connections
psql -c "SELECT count(*) FROM pg_stat_activity WHERE state = 'active';"

If the count grows rapidly and new requests fail, you've hit a connection limit.

Testing for Cache Eviction Patterns

Test cache behavior under load:

# Simulate cache warm-up and eviction
hey -n 1000 -c 50 https://api.example.com/cached-endpoint

If you see increased latency or cache miss rates, the cache is not performing as expected.

Testing for Load Balancer Behavior

Test load balancer response under stress:

# Use hey to test with multiple endpoints
hey -n 1000 -c 100 https://api.example.com/endpoint

If some instances return errors while others respond normally, you've found a load balancer or instance issue.

Testing for Service Mesh Limits

If using service mesh, test its behavior under load:

# Use wrk to simulate traffic
wrk -t5 -c100 -d60s https://api.example.com/endpoint

If latency increases or errors appear, you've found a service mesh constraint.

Testing for Sidecar Container Limits

If using sidecars, monitor their resource usage:

# Monitor sidecar pods with kubectl top
kubectl top pods -n production --selector app=sidecar

If the sidecar consumes all its allocated resources, it may cause request failures.

Testing for Ingress Controller Limits

Test ingress controller behavior under load:

# Use hey to test ingress endpoint
hey -n 1000 -c 50 https://api.example.com/ingress-endpoint

If you see timeouts or errors, your ingress may be limiting traffic.

Testing for Reverse Proxy Limits

Monitor reverse proxy behavior:

# Use curl with verbose output to trace proxy behavior
curl -v https://api.example.com/proxied-endpoint

If headers are truncated or connections drop, you've found a reverse proxy limit.

Testing for TLS Handshake Overhead

Test TLS connection overhead under load:

# Use openssl s_client to test TLS handshake
openssl s_client -connect api.example.com:443

If handshake times increase under load, TLS may be a bottleneck.

Testing for Content Encoding Limits

Test with compressed content:

# Send request with compression
curl -H "Accept-Encoding: gzip" https://api.example.com/compressed-endpoint

If response sizes are unexpectedly large or decompression fails, you've found encoding issues.

Testing for CORS Behavior

Test cross-origin requests under load:

# Use curl with custom headers
curl -H "Origin: https://app.example.com" \
     -H "Access-Control-Request-Method: POST" \
     https://api.example.com/cors-endpoint

If CORS headers are missing or response times increase, you've found a CORS issue.

Testing for Authentication Overhead

Test authentication under load:

# Use hey with authentication header
hey -H "Authorization: Bearer token123" \
    -n 1000 -c 100 https://api.example.com/authenticated-endpoint

If response times increase or auth fails, you've identified an auth bottleneck.

Testing for Authorization Checks

Test authorization behavior under load:

# Use hey with various user roles
hey -H "X-User-ID: 12345" \
    -n 1000 -c 100 https://api.example.com/authorized-endpoint

If permissions are checked slowly or fail, you've found an authorization constraint.

Testing for Cache Hit/Miss Patterns

Monitor cache behavior during load:

# Use hey to generate cache patterns
hey -n 1000 -c 50 https://api.example.com/cache-endpoint

If cache hit rates drop or miss rates increase, you've found a caching issue.

Testing for CDN Behavior

Test CDN performance under load:

# Use curl to test CDN endpoint
curl -H "Accept: application/json" \
     https://cdn.example.com/static-file.js

If response times are inconsistent or files fail to load, you've identified a CDN constraint.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.