Api Pagination Patterns

Production engineering guide for api pagination patterns covering patterns, implementation strategies, and operational best practices.

API pagination is essential when a resource returns more data than can fit in a single HTTP response. It enables efficient retrieval of large datasets by splitting results into discrete pages, reducing latency, memory use, and network overhead. This pattern matters when clients fetch collections—like user lists, order histories, or event streams—where the total count exceeds tens of thousands, and clients need to navigate, filter, or load more data incrementally.

When a client needs to traverse a large dataset—such as fetching all users in a system with 100,000 records—pagination allows it to make multiple requests, each returning a subset of data, with a clear way to navigate forward and backward.

The most common approach is offset-based pagination, where each request includes an offset and limit parameter:

GET /users?limit=25&offset=50

The server returns a list of items and metadata:

{
  "data": [
    { "id": 51, "name": "Alice" },
    { "id": 52, "name": "Bob" },
    ...
  ],
  "meta": {
    "total": 102432,
    "limit": 25,
    "offset": 50,
    "pages": 4098,
    "next": "/users?limit=25&offset=75",
    "previous": "/users?limit=25&offset=25"
  }
}

Clients use next and previous URLs to navigate. A client might start at ?limit=100&offset=0, then follow next links until no next URL exists.

This pattern is simple and effective, but has a critical flaw: inefficient for deep pagination. With offset=10000, the database must scan 10,000 rows before returning the 100 requested. As the dataset grows, each offset request becomes slower, and the cost of fetching the 100th page increases linearly with the number of preceding rows.

Efficient pagination with cursors

To overcome offset inefficiency, cursor-based pagination uses a cursor—a unique value that marks the position in the dataset. The cursor is typically a key (e.g., primary key) or a composite value (e.g., (created_at, id)).

The client requests data starting from a cursor:

GET /users?cursor=MTAwMDA=

The server returns results and a new cursor for the next page:

{
  "data": [
    { "id": 10001, "name": "Charlie" },
    { "id": 10002, "name": "Diana" }
  ],
  "meta": {
    "cursor": "MTAwMDI=",
    "has_next": true,
    "total": 102432
  }
}

The client uses the returned cursor as the cursor parameter in the next request. The server uses this cursor to locate the starting point efficiently.

A common implementation uses a last-seen ID as the cursor. The client starts with cursor=null or cursor=0:

GET /users?cursor=10002

The backend uses a query like:

SELECT * FROM users
WHERE id > $1
ORDER BY id ASC
LIMIT $2;

This is efficient even when the dataset grows to millions of rows. The cursor can be encoded as a base64 or JSON string for transport:

{
  "cursor": "eyJpZCI6IDEwMDAyLCJjb3JlX2NvbnRyb2wiOiB0cnVlLCAiY3JlYXRlZF9hdCI6ICIyMDI0LTAxLTAxVDAwOjAwOjAwWiJ9"
}

The client can also use compound cursors to handle ordering with ties:

GET /users?cursor=10000,2024-01-01T00:00:00Z

The server applies a WHERE clause on multiple fields:

SELECT * FROM users
WHERE (id > $1 OR (id = $1 AND created_at > $2))
ORDER BY id ASC, created_at ASC
LIMIT $3;

This ensures consistent ordering even when multiple records have the same id or created_at.

Handling large datasets with streaming

For clients that need to process all data in real time—like data pipelines or dashboards—streaming pagination provides a continuous flow of data.

The client issues a request with a stream=true flag:

GET /events?stream=true&limit=1000

The server responds with a chunked transfer encoding HTTP response, where each chunk contains a page of data:

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/json

{"data": [{"id": 1, "type": "purchase"}], "meta": {"cursor": "1", "has_next": true}}
{"data": [{"id": 2, "type": "login"}], "meta": {"cursor": "2", "has_next": true}}
{"data": [{"id": 3, "type": "update"}], "meta": {"cursor": "3", "has_next": false}}

Each JSON object is a page. The client reads the stream until the final chunk, where has_next is false.

This pattern enables real-time processing with minimal latency. A client can begin processing the first few events while the server streams the rest. The stream can be resumed from any point using a cursor.

The client can also pause and resume the stream by sending a PATCH request to update the cursor:

PATCH /events/stream
Content-Type: application/json

{
  "cursor": "5432"
}

The server adjusts its position and resumes streaming from that cursor.

Dealing with failure and consistency

Pagination patterns fail silently in production. The most common failure is inconsistent state between pages.

When a client fetches ?limit=100&offset=0, and during the response, a new user is created, the client might miss that user or see duplicates. This happens because the offset is calculated at the start of the request, but data can change during execution.

To address this, staleness-aware pagination includes a timestamp or version in the response:

{
  "data": [...],
  "meta": {
    "timestamp": "2024-01-15T12:34:56Z",
    "version": 12345,
    "next": "/users?limit=100&offset=100&version=12345"
  }
}

The client can validate the consistency of the dataset by comparing the version with a known state. If a new version arrives, the client can re-fetch or merge changes.

Another failure mode is missing or malformed cursors. A cursor might be a string, but the server expects a specific format—e.g., base64-encoded JSON. A client using a simple number-based cursor might send cursor=10000, but the server expects cursor=MTAwMDA=. The result is a 400 Bad Request with a clear error message:

{
  "error": "invalid_cursor",
  "message": "Cursor must be a valid base64-encoded JSON object",
  "expected": "base64(json)",
  "received": "10000"
}

Clients should validate cursor format and handle mismatched types.

Combining pagination strategies

In practice, APIs combine multiple patterns. A client might start with offset-based pagination for simplicity and then switch to cursor-based pagination for deeper navigation.

The API can support both, with the cursor parameter taking precedence over offset and limit. If a client sends both, the server applies cursor logic and ignores offset.

GET /orders?cursor=12345&limit=50&offset=200

The server:

  1. Uses cursor to find the starting point.
  2. Applies limit to fetch up to 50 items.
  3. Ignores offset (or uses it as a fallback if cursor is missing).

Clients can also prefetch multiple pages:

GET /products?limit=50&cursor=1000

The client receives:

The client can preload the next two pages in the background.

Client-side pagination patterns

Clients often implement infinite scroll or load-more interfaces. The client fetches the first page, then triggers a fetch when the user scrolls to the end.

A common pattern is lazy loading:

let currentPage = 0;
let totalPages = 1;
let hasMore = true;

async function loadMore() {
  if (!hasMore) return;

  const response = await fetch(`/users?limit=25&offset=${currentPage * 25}`);
  const data = await response.json();

  // Append to DOM
  data.data.forEach(user => appendUser(user));

  currentPage++;
  hasMore = data.meta.next !== null;
}

The client listens to scroll events and calls loadMore when the user reaches 80% of the page.

For better UX, clients can cache paginated results in memory or local storage. Each page is keyed by its offset or cursor, and the client can retrieve a page without a network request.

Error handling and debugging

Pagination APIs must return clear error messages when requests fail. A client should be able to diagnose issues with minimal effort.

Common errors include:

The server should include a trace-id or request-id in the response for debugging:

{
  "error": "invalid_cursor",
  "message": "Cursor 'abc123' not found in users dataset",
  "request_id": "req-1234567890",
  "timestamp": "2024-01-15T12:34:56Z"
}

Clients can log these IDs when an error occurs and correlate them with backend logs.

Summary of key practices

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.