Streaming LLM Responses | The Garnet Wiki

Streaming LLM responses deliver token-by-token output from a large language model as it generates text, enabling real-time, incremental feedback. This is essential when users expect immediate interaction, such as in chat applications, live code assistants, or interactive data exploration systems. The key signal that streaming is required is the need to begin rendering content before the full response is ready — when latency is more critical than final accuracy, and when users are actively engaged and expect to see progress.

Core Mechanism: Streaming over HTTP

Streaming LLM responses is fundamentally an HTTP-based pattern where the server sends chunks of text progressively in a single response stream. The client opens a connection and begins receiving data as the model computes its output.

The HTTP response includes a Content-Type header specifying text/event-stream (or application/json; charset=utf-8 for JSON-streamed responses), and the body is a sequence of newline-separated events. Each event may include a data: field, optionally prefixed by event:, id:, and retry:.

Basic Streaming with curl

To test or debug an LLM streaming endpoint, use curl with the --stream flag:

curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Explain quantum entanglement in simple terms.", "stream": true}' \
  --stream \
  http://localhost:8080/v1/completions

This returns a stream of lines like:

data: {"id":"cmpl-123","object":"text_completion","created":1647291600,"model":"gpt-3.5-turbo","choices":[{"index":0,"text":"Quantum entanglement is","finish_reason":"length"}], "stream":true}

data: {"id":"cmpl-123","object":"text_completion","created":1647291600,"model":"gpt-3.5-turbo","choices":[{"index":0,"text":"Quantum entanglement is a phenomenon where two or more particles become linked in such a way that the state of one particle instantly influences the state of the other, even when separated by large distances."}], "stream":true}

data: {"id":"cmpl-123","object":"text_completion","created":1647291600,"model":"gpt-3.5-turbo","choices":[{"index":0,"text":"Quantum entanglement is a phenomenon where two or more particles become linked in such a way that the state of one particle instantly influences the state of the other, even when separated by large distances. This effect was first proposed by Einstein, Podolsky, and Rosen in 1935, and is often referred to as the EPR paradox."}], "stream":true}

data: {"id":"cmpl-123","object":"text_completion","created":1647291600,"model":"gpt-3.5-turbo","choices":[{"index":0,"text":"Quantum entanglement is a phenomenon where two or more particles become linked in such a way that the state of one particle instantly influences the state of the other, even when separated by large distances. This effect was first proposed by Einstein, Podolsky, and Rosen in 1935, and is often referred to as the EPR paradox. In practical terms, entangled particles can be used to transmit information faster than classical methods, enabling quantum computing and quantum cryptography."}], "stream":true}

data: {"id":"cmpl-123","object":"text_completion","created":1647291600,"model":"gpt-3.5-turbo","choices":[{"index":0,"text":"Quantum entanglement is a phenomenon where two or more particles become linked in such a way that the state of one particle instantly influences the state of the other, even when separated by large distances. This effect was first proposed by Einstein, Podolsky, and Rosen in 1935, and is often referred to as the EPR paradox. In practical terms, entangled particles can be used to transmit information faster than classical methods, enabling quantum computing and quantum cryptography."}], "stream":true}

The stream ends when the server closes the connection. Clients must handle partial payloads and reconstruct the final output by concatenating data fields.

Client-Side Handling in JavaScript

In a web application, the fetch API supports streaming via ReadableStream:

const response = await fetch('/api/v1/completions', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    prompt: 'Summarize this document.',
    stream: true,
    temperature: 0.7,
  }),
});

const reader = response.body.getReader();
const decoder = new TextDecoder('utf-8');

let result = '';

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  const chunk = decoder.decode(value);
  const lines = chunk.split('\n');
  for (const line of lines) {
    if (line.startsWith('data:')) {
      const data = JSON.parse(line.slice(5));
      result += data.choices[0].text;
      document.getElementById('output').textContent = result;
    }
  }
}

The data: prefix is required. If the client expects plain text instead of JSON, the server may emit text/plain with Content-Type: text/plain; charset=utf-8 and send raw text, line-by-line:

Quantum entanglement is
a phenomenon where two or more particles become linked
in such a way that the state of one particle instantly influences
the state of the other, even when separated by large distances.
This effect was first proposed by Einstein, Podolsky, and Rosen in 1935,
and is often referred to as the EPR paradox.

Configuration and Tuning

Streaming is not just a mode; it is a set of configuration knobs that affect performance, resource usage, and user experience.

Streaming Flags and Parameters

When calling an LLM API, the client may include:

stream: true — enables streaming mode
max_tokens: 1024 — maximum number of tokens to generate
temperature: 0.7 — controls randomness
top_p: 0.9 — nucleus sampling
frequency_penalty: 0.5 — discourages repeated tokens
presence_penalty: 0.3 — encourages new topics

The server must process these parameters and generate tokens incrementally. The stream flag is a boolean, but it may be accompanied by a stream_options object:

{
  "prompt": "Explain transformers in NLP.",
  "stream": true,
  "stream_options": {
    "include_usage": true,
    "include_prompt": true,
    "chunk_size": 1024,
    "delimiter": "\n"
  }
}

The chunk_size controls how many bytes are sent per network packet. A value of 1024 is common. The delimiter field specifies the character used to split the stream into chunks (e.g., newline, space, or a custom marker).

Chunking and Delimiter Strategy

When delimiter is set, the server sends chunks whenever a delimiter is encountered. For example, setting delimiter: "\n" means every newline triggers a flush.

This is useful for interactive systems where the user expects a new paragraph or line to appear immediately:

data: {"id":"cmpl-456","choices":[{"text":"Transformers are a neural network architecture introduced in 2017."}], "finish_reason":"length"}
data: {"id":"cmpl-456","choices":[{"text":"They are designed to handle sequential data like text using self-attention mechanisms."}], "finish_reason":"length"}
data: {"id":"cmpl-456","choices":[{"text":"This allows the model to weigh the importance of different parts of the input when processing each token."}], "finish_reason":"length"}

Clients can render each chunk as a new paragraph. The include_prompt flag causes the model to send the prompt as part of the stream, so the client can display the input text while the model works.

Error Handling and Robustness

Streaming is fragile. A network interruption can leave the client with partial data. The server may fail mid-stream, and the client must handle:

Connection timeout (read_timeout: 30s)
retry: in the event stream to define reconnection intervals
id: to track which response stream a chunk belongs to

If the client reconnects, it may send an id field to resume from a specific point:

data: {"id":"cmpl-789","choices":[{"text":"The self-attention mechanism computes a weighted sum of all input tokens to each position."}], "stream":true}
data: {"id":"cmpl-789","choices":[{"text":"This enables the model to capture long-range dependencies effectively."}], "stream":true}

A client that lost the connection after the first chunk can resume by sending a Last-Event-ID header:

GET /api/v1/completions
Host: api.example.com
Last-Event-ID: cmpl-789

The server responds with all events from id: cmpl-789 onward, ensuring no data loss.

Common Failure Modes

1. Misaligned Event Parsing

A common error arises when the client assumes data: is always prefixed with data: but the server sends data: (with a space after the colon):

data: {"text":"Hello"}
data:  {"text":"World"}  <!-- note the space after 'data:'

The parser reads the second line as {"text":"World"} but the first data: is missing. The client receives a single JSON object {"text":"World"} but expects a stream of events.

The solution: normalize the data: field by trimming whitespace and splitting on data: followed by any whitespace.

2. Incomplete JSON in Stream

When the server sends JSON chunks that are not complete, the client may receive:

data: {"id":"cmpl-123","choices":[{"text":"Partial

The JSON is cut off mid-string. The client must accumulate data until it parses a complete JSON object. The decoder in JavaScript must be configured to handle partial input:

const decoder = new TextDecoder('utf-8');
let accumulated = '';

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  const chunk = decoder.decode(value);
  accumulated += chunk;

  let pos = 0;
  while (pos < accumulated.length) {
    const start = pos;
    if (accumulated[pos] !== 'd') break;

    const line = accumulated.slice(pos, accumulated.indexOf('\n', pos));
    if (line.startsWith('data:')) {
      const data = line.slice(5);
      const json = data.trim();
      try {
        const parsed = JSON.parse(json);
        // Process parsed event
        pos += line.length + 1;
      } catch (e) {
        // Incomplete JSON — break and wait for more data
        break;
      }
    } else {
      pos += 1;
    }
  }
}

3. Overlapping and Duplicated Chunks

When the chunk_size is small and the network is slow, the client may receive overlapping chunks. For example, the server sends 1KB chunks, but the client reads 512 bytes at a time:

Chunk 1: {"id":"cmpl-123","choices":[{"text":"Hello, world!"}]
Chunk 2: {"id":"cmpl-123","choices":[{"text":"Hello, world! How are you?"}]

The client may process both chunks, but the second one includes the first, leading to duplicate output.

The solution: use id to deduplicate, or use a sequence_number field in the event payload. The client can buffer and merge chunks based on id and sequence_number.

4. Streaming with Delayed Final Output

The model may generate a response and then return a final finish_reason event:

{
  "id": "cmpl-123",
  "object": "text_completion",
  "created": 1647291600,
  "model": "gpt-3.5-turbo",
  "choices": [
    {
      "index": 0,
      "text": "Quantum entanglement is a phenomenon where two or more particles become linked...",
      "finish_reason": "stop"
    }
  ],
  "stream": true
}

Clients must wait for the final event to know the response is complete. If the client renders the stream and then adds a “Done” indicator, it should listen for finish_reason to be stop, length, or max_tokens.

5. Server-Side Buffering and Flush Behavior

The server may buffer output and send chunks only when a threshold is reached. If the model generates tokens slowly, the client sees long delays between chunks. The server must flush the buffer periodically:

Every 500ms
After every 50 tokens
When the buffer reaches 2KB

Clients should expect irregular intervals and handle both small and large chunks. A common mistake is assuming the stream is perfectly paced, but in reality, the model may generate tokens at variable rates due to attention computation, cache misses, or GPU scheduling.

Best Practices

Always include Content-Type: text/event-stream and use data: prefix.
Use id to enable resume-from-last-event and deduplication.
Set chunk_size appropriately: 1024 is a good default, but tune based on network latency and client rendering.
Include finish_reason in the final event to signal completion.
Use retry: to define reconnection intervals, especially in unreliable networks.
Handle partial and malformed JSON gracefully.
Render the stream incrementally, and use include_prompt and include_usage for richer feedback.

Streaming LLM responses is a pattern that is both simple and subtle. It requires attention to detail in both client and server, but once implemented, it transforms user experience from "waiting" to "interacting."

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.