Rate Limiting and API Throttling: Protecting Services at Scale

Implementation patterns for rate limiting and API throttling — covering algorithms, distributed rate limiting, quota management, and graceful degradation.

Rate limiting and API throttling are essential techniques for protecting backend services under load, ensuring predictable performance, and preventing resource exhaustion across distributed systems. They apply when a service receives more requests than it can handle efficiently—common in microservices, public APIs, and event-driven architectures—where unbounded traffic can overwhelm databases, queues, and compute instances.

Controlling Request Volume at the Edge

To enforce rate limits, a service must track how many requests a client makes within a defined time window, and act when that volume exceeds a threshold. The most common approach is the token bucket algorithm, implemented in the edge layer (typically a reverse proxy or API gateway). A client receives a fixed number of tokens per time interval (e.g., 100 tokens per minute). Each request consumes one token. When tokens are depleted, the service rejects new requests until more tokens are added.

A typical request includes a RateLimit-Remaining header and a Retry-After header when throttled:

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
RateLimit-Limit: 100
RateLimit-Remaining: 0
Retry-After: 60
{
  "error": "Rate limit exceeded",
  "details": "Maximum 100 requests per minute per client"
}

Clients should respect the Retry-After value, either by retrying after the specified seconds or using exponential backoff. The RateLimit-Limit header communicates the total allowed requests per window.

Configuring a Token Bucket with a Standard API Gateway

Using a standard API gateway (e.g., NGINX, Traefik, or Envoy), a rate limit is configured via a declarative configuration block. In NGINX, this is done in the http or location block:

location /api/v1/users {
    limit_req zone=user_api burst=20 nodelay;
    limit_req_status 429;
    limit_req_log_level warn;

    # Define the zone: shared memory between workers
    limit_req_zone $binary_remote_addr zone=user_api:10m rate=100r/s;

    proxy_pass http://backend-users;
}

This configuration:

The binary_remote_addr key ensures that clients are identified by IP address (or IPv6 address), enabling per-client throttling.

Per-Client Throttling with Headers and Tokens

When clients are authenticated, throttling can be applied per user, per API key, or per client ID. A common pattern uses an X-API-Key header to identify the client:

GET /api/v1/orders HTTP/1.1
Host: api.example.com
X-API-Key: a1b2c3d4e5f6
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...

The gateway extracts the API key from the header and uses it as a key in a shared bucket. In Envoy, this is achieved with the rate_limit filter:

filters:
  - name: envoy.filters.http.rate_limit
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.rate_limit.v3.RateLimit
      domain: "api"
      failure_mode: true
      stage: 0
      descriptors:
        - key: api_key
          value: "api_key"
          entries:
            - key: "api_key"
              value: "a1b2c3d4e5f6"
              rate_limit:
                unit: second
                requests_per_unit: 100
                bucket_size: 100

This config applies a rate limit of 100 requests per second, with a bucket size of 100 tokens, to each unique API key. The failure_mode: true setting means that if the rate limit service is unreachable, the request proceeds and the limit is applied locally.

Handling Bursty Traffic with Dynamic Thresholds

For services with uneven traffic—such as event-driven systems or real-time dashboards—static rate limits are insufficient. A client may send 50 requests in 10 seconds, then idle for a minute, only to send 300 more in the next 30 seconds.

To handle this, the system must support variable windowing: fixed, sliding, and cumulative windows.

The fixed window is simple but can allow burst spikes:

But this allows up to 200 requests in a single minute: 100 in the old window and 100 in the new.

A sliding window reduces this by tracking individual request timestamps. In practice, this is implemented with Redis using a ZSET for each client:


ZADD user_requests:a1b2c3d4e5f6 1700000000 "2023-11-15T12:00:00Z"
ZADD user_requests:a1b2c3d4e5f6 1700000030 "2023-11-15T12:00:30Z"
ZADD user_requests:a1b2c3d4e5f6 1700000100 "2023-11-15T12:01:40Z"

To count requests in the last 60 seconds:

ZRANGEBYSCORE user_requests:a1b2c3d4e5f6 1700000000 1700000600

This returns all requests within the sliding window. The count is compared against the allowed rate. The system can be backed by a Redis Lua script to atomically check and update the limit.

Implementing Hierarchical Throttling

In multi-tiered systems, different rate limits apply at different layers. For example:

To enforce this, the system must evaluate multiple descriptors in sequence.

In Envoy, this is achieved by nesting descriptors:

descriptors:
  - key: global
    rate_limit:
      unit: second
      requests_per_unit: 10000
  - key: api_key
    value: "a1b2c3d4e5f6"
    rate_limit:
      unit: minute
      requests_per_unit: 1000
  - key: user_id
    value: "u12345"
    rate_limit:
      unit: minute
      requests_per_unit: 100

The service evaluates each descriptor in order. If any fails (e.g., the user has sent 101 requests in a minute), the request is throttled. The response includes the X-RateLimit headers for each level:

HTTP/1.1 429 Too Many Requests
X-RateLimit-Limit: 10000
X-RateLimit-Remaining: 9999
X-RateLimit-Reset: 60
X-RateLimit-User-Limit: 100
X-RateLimit-User-Remaining: 99
X-RateLimit-User-Reset: 58
Retry-After: 60
{
  "error": "Rate limit exceeded",
  "details": [
    "Global limit exceeded: 10,000 requests per second",
    "User u12345 exceeded 100 requests per minute"
  ]
}

This allows clients to understand which layer caused the throttling and plan retries accordingly.

Scaling the Rate Limiting Engine

The rate limit engine must scale with traffic and be resilient to failure. A single Redis instance can serve as a backing store for token bucket state, but it becomes a bottleneck under high load.

To scale, the system uses distributed rate limiters with consistent hashing or sharding. Each client key is hashed to a Redis node:

# Hash client key to node using consistent hashing
KEY = "api_key:a1b2c3d4e5f6"
NODE = hash(KEY) % 8

Each node manages a subset of the client buckets. To ensure high availability, Redis is run in a cluster with automatic failover.

When a rate limiter service is unavailable, clients can fall back to local rate limiting. The gateway maintains a local bucket for each client, using memory or a local cache (e.g., in-memory LRU). The fallback process is:

  1. Attempt to contact the global rate limiter service (e.g., via gRPC).
  2. If unreachable, use local bucket.
  3. After successful call, update the global state asynchronously.

This pattern is implemented in the rate_limit filter of Envoy with async: true:

typed_config:
  "@type": type.googleapis.com/envoy.extensions.filters.http.rate_limit.v3.RateLimit
  async: true
  domain: "api"
  descriptors:
    - key: api_key
      value: "api_key"
      rate_limit:
        unit: minute
        requests_per_unit: 100

The async: true setting enables non-blocking calls to the rate limiter service, allowing the gateway to serve requests even when the limiter is down.

Common Pitfalls and Failure Modes

Monitoring and Observability

A rate-limited system must be observable. Key metrics include:

The system logs every rate limit decision:

{
  "timestamp": "2023-11-15T12:00:00Z",
  "client_id": "a1b2c3d4e5f6",
  "api_key": "a1b2c3d4e5f6",
  "user_id": "u12345",
  "request_id": "req-12345",
  "rate_limit": {
    "domain": "api",
    "limit": 100,
    "remaining": 17,
    "window_start": "2023-11-15T12:00:00Z",
    "window_end": "2023-11-15T12:01:00Z"
  },
  "throttled": true,
  "retry_after": 60
}

These logs are consumed by monitoring tools (Prometheus, Grafana, Loki) to visualize load patterns and detect anomalies.

Final Notes: The Art of the Throttled Response

A well-implemented rate limiting strategy does more than just reject requests. It communicates clearly, adapts to traffic, and provides context for clients. The 429 Too Many Requests response is the cornerstone. It must include:

Clients should not only retry but also:

When rate limiting is done at scale, it becomes a self-healing layer: it prevents cascading failures, enables graceful degradation, and provides a foundation for service-level agreements (SLAs).

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.