Performance Testing & Load Engineering

Design and run performance tests. Covers load testing, stress testing, soak testing, k6, Gatling, JMeter, performance budgets, and interpreting results for capacity planning.

Performance Testing & Load Engineering is the practice of measuring, validating, and tuning system behavior under realistic workloads. It matters when a service must respond predictably under sustained load, when latency spikes cause user dissatisfaction, or when a new release risks overwhelming infrastructure. This is not exploratory testing; it is engineering the system’s response to demand, using repeatable, measurable, and automated workloads.

Defining the Performance Profile

A performance profile captures the expected behavior of a system under defined load. It answers: *How many requests per second can the system handle? What is the 95th percentile latency? When does it degrade? At what point does it fail?*

The profile begins with a workload definition: a sequence of requests that simulate real user behavior. Each request is a transaction, composed of one or more HTTP operations, database queries, message sends, or external API calls. These transactions are grouped into a test scenario—a set of concurrent users, each executing a sequence of transactions over time.

For example, a typical e-commerce profile includes:

Each transaction is timed, and the results are aggregated to compute metrics: throughput (requests per second), latency (time from start to response), error rates, and resource utilization (CPU, memory, I/O).

Running a Load Test

To run a load test, you need a load generator—a service that spawns virtual users and emits requests at defined rates. The generator must:

A minimal but effective setup uses hey (from github.com/rakyll/hey), a command-line tool for HTTP load testing.

hey -n 10000 -c 100 -z 10m \
  -m POST \
  -H "Content-Type: application/json" \
  -d '{"user": "alice", "action": "login"}' \
  http://api.example.com/auth/login

This command:

The output includes:

When the test fails, the error threshold is not just "failed requests," but a failure condition—e.g., 95th percentile latency > 200ms or error rate > 1%.

Designing Realistic Workloads

The key to effective load engineering is realism. A test that sends identical requests every second is brittle. Real users vary: some wait 10 seconds between actions, others click rapidly, and some drop off mid-flow.

To simulate this, use ramp-up profiles and think times.

A ramp-up profile defines how virtual users are introduced over time. For example:

In hey, this is achieved with a duration-based script:

hey -n 20000 -c 100 -z 15m \
  -t 60s \
  -m POST \
  -d '{"user": "alice", "action": "search"}' \
  http://api.example.com/products

The -t 60s flag introduces one user every 60 seconds (i.e., 1 user per minute) during the test, creating a steady ramp-up.

For think times, use scripted scenarios with k6 (from k6.io), a JavaScript-based load testing tool.

import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
  vus: 100,
  duration: '10m',
  thresholds: {
    http_req_duration: ['p(95)<200'],
    'http_req_error_rate': ['lt=0.01'],
  },
};

export default function () {
  // Login
  let res = http.post('http://api.example.com/auth/login', JSON.stringify({
    username: 'testuser',
    password: 'pass123',
  }));
  check(res, { 'login status is 200': (r) => r.status === 200 });

  // Search
  sleep(10); // think time
  res = http.get('http://api.example.com/products?query=shoes');
  check(res, { 'search returns 200': (r) => r.status === 200 });

  // Add to cart
  sleep(5);
  res = http.post('http://api.example.com/cart', JSON.stringify({
    product_id: 123,
    quantity: 2,
  }));
  check(res, { 'add to cart success': (r) => r.status === 201 });

  // Checkout
  sleep(15);
  res = http.post('http://api.example.com/checkout', JSON.stringify({
    shipping: 'standard',
    payment: 'credit',
  }));
  check(res, { 'checkout success': (r) => r.status === 201 });
}

This script:

The k6 tool runs this script, collects metrics, and produces structured output in JSON or CSV for integration into dashboards.

Measuring and Validating System Behavior

A load test is only valuable when its results are interpreted and used to validate the system’s performance behavior. The primary metrics are:

To validate a system, define thresholds—explicit, measurable conditions that the system must meet.

For example, a profile for a payment service might require:

These thresholds are checked automatically. A test fails if:

Failure is not just a test log; it is a failure condition that can be evaluated by a CI/CD pipeline.

Tuning the System

When a load test reveals degradation, engineers tune the system. Tuning begins with diagnostic data collected during the test.

Key diagnostic signals include:

Use curl to probe endpoints during a test:

curl -v -H "X-Request-ID: 12345" \
  http://api.example.com/products?query=shoes

The -v flag reveals:

For database performance, use EXPLAIN ANALYZE in psql:

EXPLAIN ANALYZE
SELECT * FROM orders
WHERE created_at > '2024-01-01'
  AND status = 'shipped'
ORDER BY total DESC
LIMIT 10;

This returns:

A slow query with high cost and actual rows far from estimated rows indicates a missing or inefficient index.

When a system fails under load, the root cause is often resource saturation:

In Kubernetes, monitor with kubectl top pods and kubectl get hpa (Horizontal Pod Autoscaler):

kubectl top pods -n payment-service
kubectl get hpa -n payment-service

If CPU is high and HPA shows no scaling, the problem is likely insufficient replicas or inadequate resource requests in the pod spec.

Handling Failure Modes

Load engineering exposes silent failures that do not appear in development or staging.

A common failure is slow response due to connection exhaustion. The system starts with 100 concurrent connections, but the number of active connections grows beyond 100. This happens when:

Another failure: latency spike due to garbage collection. A test runs for 15 minutes, and the 95th percentile latency jumps from 120ms to 300ms after 7 minutes. The cause is a full GC cycle every 7 minutes, triggered by a large heap and long-running threads.

To detect this, enable GC logging in the JVM:

jstat -gcutil 12345 1s 10

This command outputs GC statistics every second for 10 seconds, showing:

A spike in S0 and S1 space indicates frequent minor collections, while high O and M usage with long pause times suggest major GCs.

Integrating Load Tests into CI/CD

Load tests must run not just in staging, but in every CI/CD pipeline. They are not one-off scripts; they are regression checks.

In a CI/CD system, a new commit triggers:

A Makefile might include:

.PHONY: test performance

test: unit integration

performance: test
	@echo "Running load test..."
	@k6 run tests/performance.js \
		--env production \
		--thresholds 'http_req_duration[p(95)]<200' \
		--thresholds 'http_req_error_rate<0.01'
	@echo "Load test completed."

The pipeline fails if:

The test output is stored as a test artifact—a JSON file, CSV, or HTML report—available for review.

Common Pitfalls

curl -s http://api.example.com/products | jq '.items | length > 0'

Performance Testing & Load Engineering is not a series of isolated tests—it is a continuous loop of measurement, validation, tuning, and integration. The system evolves not just with new features, but with predictable, measurable performance.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.