API Rate Limiting Impl | The Garnet Wiki

API Rate Limiting Impl

API rate limiting implementation is the server-side enforcement of request quotas per client identity, API key, or IP address. It matters when you need to protect backend stability, enforce service-level quotas, prevent abuse, or throttle misbehaving consumers without requiring code changes in every handler. The implementation sits between the API gateway or ingress layer and your application logic, and its configuration directly affects which requests succeed and which are rejected with a 429 Too Many Requests response. Getting it right requires understanding the algorithm, the identity model, and how the limits interact with downstream services.

Checking Enforcement from the Client

The fastest way to verify that rate limiting is active is to probe the API with curl and inspect the response headers and status. A typical limited request returns:

curl -i https://api.example.com/resource

HTTP/1.1 429 Too Many Requests
Date: Mon, 24 Feb 2025 18:03:12 GMT
Content-Type: application/json
Retry-After: 3600
Cache-Control: no-store
Content-Length: 73

{"error":"rate limit exceeded","retry_after":3600}

If the response is 200 OK, check for rate-limit diagnostic headers that some platforms emit on successful requests: RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset. Their absence does not guarantee that limiting is disabled; the server may be configured to emit them only on rejection, or a downstream proxy may be stripping them.

When headers are missing and you suspect limiting is in effect, observe request patterns over time. A sudden drop in success rate without changes to authentication or payload size often indicates that a quota has been reached. If you control the server, ensure the limiting module is enabled and that its response body and headers are not blocked by a WAF or reverse proxy.

Configuring Server-Side Thresholds

Most rate limiters rely on one of three algorithmic patterns: token bucket, sliding window, or leaky bucket. The configuration always expresses two core parameters—a rate and a burst (or window) size.

Rate is the sustained throughput, expressed as requests per second (or per minute/hour depending on the deployment).
Burst (token bucket) or window (sliding window) is the maximum number of requests allowed above the sustained rate before rejection occurs.

These values are typically read from the service’s runtime configuration. In many systems they are set via environment variables, a Kubernetes ConfigMap, or a service-mesh policy file. The exact key names vary by platform; you will need to map the conceptual rate and burst to whatever your deployment system expects. A common pitfall is setting the burst too low: legitimate traffic spikes (e.g., a batch job startup or a webhook retry storm) are rejected even though the long‑term average is well within the rate. Conversely, a burst that is too large defeats the purpose of the limit, allowing a brief surge to flood the backend.

When multiple service instances run behind a load balancer, each instance maintains its own counter unless a shared store (Redis, Memcached, or a distributed lock service) is used. Per‑instance limits add together, which can unintentionally exceed the intended global quota. If you observe rejections that seem higher than the configured rate, verify whether the limit is enforced per host or globally.

Handling 429 Responses in Client Code

When a request is rejected, the server returns HTTP/1.1 429 Too Many Requests. The response body commonly contains a machine‑readable error, but its shape is not standardized. A typical JSON payload looks like:

{"error":"rate limit exceeded","retry_after":3600}

The Retry-After header, when present, indicates the number of seconds the client should wait before retrying. If the header is absent, a conservative default is to back off exponentially, doubling the wait time after each consecutive 429, up to a maximum cap (e.g., 300 seconds).

Clients should treat 429 as a non‑idempotent guard unless the underlying operation is explicitly marked safe to repeat. Idempotent requests (GET, HEAD, OPTIONS on read‑only resources) may be retried after the Retry-After interval. Non‑idempotent operations (POST, PUT, DELETE) should not be retried automatically; instead, surface the error to the user and log the event for ops review. Many frameworks provide a hook or middleware that injects a Retry-After–aware delay before the next attempt, which is preferable to hard‑coding sleep times.

If the API strips the body and only returns the status line and Retry-After, design your client to read the header and apply the same backoff strategy. Never assume a specific body format; build parsers that tolerate missing fields and unexpected types.

Tuning the Rate Algorithm to Your Traffic Shape

The choice of algorithm shapes how bursts and sustained traffic are handled.

Token bucket – Tokens are added to a bucket at the configured rate. Each request consumes one token. If the bucket is empty, the request is rejected. The bucket can hold up to burst tokens, allowing short‑term bursts as long as the long‑term average stays within the rate. This is well‑suited for APIs where occasional spikes are expected (e.g., a dashboard refresh every few seconds).

Sliding window – Requests are counted in the most recent window seconds. Once the count exceeds the limit, new requests are rejected until the window slides enough counts out. This produces a more predictable rejection pattern for steady traffic but can be vulnerable to coordinated bursts across distributed instances, since each instance counts independently.

Leaky bucket – Requests are enqueued and then served at a constant output rate, smoothing bursty input into a steady outflow. This is useful when the downstream service has a fixed processing capacity (e.g., a message queue with a max throughput) and you want to prevent queue buildup.

If your traffic pattern includes frequent short‑lived spikes (API clients polling every few seconds, CI webhooks), a token bucket with a modest burst is usually sufficient. For workloads with a constant, predictable flow (batch ETL streams, streaming endpoints), a leaky bucket or sliding window with a long window may give tighter control. Always measure the actual request distribution in staging before locking in values.

When tuning, start with a rate that reflects the maximum healthy throughput your backend can sustain, then add a burst equal to the size of your largest expected spike. Monitor rejection counts and adjust downward if legitimate users are hit, or upward if abuse is under‑controlled.

Debugging Common Misfires

Several subtle factors can make a correctly configured limiter appear broken or, worse, silently misfire.

Clock skew – Token‑bucket refill calculations depend on elapsed time. If service nodes have unsynchronized clocks (NTP drift > 1 second), the bucket may refill faster or slower than intended, causing under‑ or over‑limiting. Verify NTP status on all nodes and consider using a single time source for rate‑limit state if your system supports it.

IP‑hash collisions – Limiting by source IP is common for quick deployment, but many users share one NAT address. A single burst from one user can throttle every other user behind the same NAT. Prefer per‑API‑key or per‑JWT‑subject identification when possible, and only fall back to IP limiting for public endpoints with no authentication layer.

Header stripping – Proxies, CDNs, or edge firewalls sometimes remove RateLimit-* or Retry-After headers. Clients that rely on these headers to implement backoff may silently retry immediately, hitting the limit again and entering a loop. Log the presence or absence of these headers in your observability pipeline, and consider embedding limit state in the response body if the header path is unreliable.

Misidentified client – Some systems limit by the API key present in the Authorization header, others by the JWT sub claim, and others by the raw IP. If a client rotates keys or uses a service account, the limiting identity may change unexpectedly, causing intermittent 429s. Log the identity field used for limiting alongside each request, and validate that the same identity consistently maps to the same bucket.

Configuration drift – After a deploy, new rate‑limit values may not take effect until the service restarts. If you push a config change via a hot‑reload mechanism, ensure the reload is atomic and that old and new values do not coexist in a way that creates a window of unexpected allowance. Pair config changes with a rolling restart or a feature‑flag gate when possible.

Production Sharp Edges

Running rate limiting in production reveals behaviors that are easy to overlook in staging.

Distributed count inconsistency – Without a shared store, two instances may each allow a burst of burst requests, collectively exceeding the intended global limit. If global enforcement is required, centralize the counter in Redis using INCR with a TTL matching the bucket duration. Be aware that this adds one round‑trip latency per request; benchmark the trade‑off against the cost of over‑throttling.

Burst amplification – A sudden influx of burst‑sized requests can flood the backend before the refill thread catches up. Pair rate limiting with circuit breaking or bulkhead isolation so that rejected requests are handled gracefully (e.g., returned a degraded response) rather than hammering an already‑strapped service.

Rate‑limit loops – A client that does not honor Retry-After may repeatedly send requests, receive 429, wait a short period, retry, and be throttled again in a tight cycle. This generates unnecessary load and can mask genuine quota issues. Alert when a single client produces more than N 429s within a rolling window; such patterns often indicate a client bug or an integration that needs redesign.

Metrics gaps – Many implementations emit counters (requests_allowed, requests_rejected) but omit distribution histograms. Without a histogram showing the percentage of requests at or near the burst cap, you won’t know if you’re frequently hitting the burst limit, suggesting the burst is too small for your traffic, or rarely hit, suggesting the rate is too low. Push these counters to your monitoring system and include quantiles if possible.

TLS termination before limiting – If your reverse proxy or API gateway terminates TLS before the rate limiter sees the request, the limiter only observes the TCP connection, not the HTTP path. This means per‑endpoint or per‑path granularity is lost; all requests over that connection share one bucket. If you need endpoint‑level limiting, ensure the limiter operates after TLS termination, or use a mechanism that extracts the path identity before the proxy layer.

Rate limiting is a mechanical pattern, not a set‑and‑forget configuration. The right algorithm, identity model, and tuning depend on your traffic shape, your downstream capacity, and how your clients behave. Test changes in a staging environment that mirrors your production identity distribution and clock configuration, push the observed limits to monitoring, and revisit the values whenever you add new consumer types or change your deployment scale.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.