Streaming Responses

The plain-language idea

An LLM doesn't generate its full response and then hand it over — it generates one token at a time, and each token depends on every token generated before it. That's a sequential process with no way to skip ahead, which means a long response can take several real seconds to finish generating in full.

Streaming sends each token (or small chunk of tokens) to the client as soon as it's generated, over a connection that stays open for the duration of the response, instead of waiting for generation to finish and sending one final payload. The most common transport for this is Server-Sent Events (SSE): a single long-lived HTTP connection the server keeps writing small chunks into, which the browser reads incrementally as they arrive.

The reason this matters isn't really about total speed — a streamed response and a non-streamed response for the same prompt take roughly the same total time to fully generate. It's about perceived responsiveness. A multi-second wait with a static, unchanging screen reads as broken or frozen, even if it's working exactly as intended. The same multi-second wait with words appearing progressively reads as fast and alive, because the user gets continuous evidence that something is happening — the system feels responsive at the same underlying completion time.

See it in a real thing: appending chunks instead of waiting for one payload

Both requests go out at the same moment — generation is sequential, so neither can show anything yet.

BackNext

// Client-side: read a streamed response chunk by chunk and append
// each piece to render state as it arrives — never wait for the whole thing.
const reader = response.body.getReader();
const decoder = new TextDecoder();
let fullText = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  fullText += decoder.decode(value, { stream: true });
  setChatMessage(fullText); // re-render with each incoming chunk
}

This is the real implication for your frontend: streaming isn't just a server-side setting you flip on and forget. Your UI has to actually consume it incrementally — appending each incoming chunk to render state as it arrives, not buffering everything and rendering once at the end. A chat UI that streams from the server but waits for the full response client-side before displaying anything gets zero UX benefit from streaming at all; the perceived-speed win only exists if the render loop updates on every chunk.

What people usually miss

Watch out

• Streaming doesn't make generation faster — it changes when you see the result. Total time-to-completion is roughly unchanged; what improves is perceived responsiveness during that time.
• The server streaming isn't enough — the client has to render incrementally too. A frontend that buffers the whole stream before updating the UI throws away the entire benefit.
• SSE is one-directional. The server pushes chunks to the client; if the client needs to send data back mid-stream (like a cancel signal), that's typically a separate mechanism, not part of the SSE channel itself.
• A stream can fail partway through. Unlike a single request that either fully succeeds or fully fails, a stream can deliver some tokens and then drop — production code needs to handle a partial, cut-off response, not just success/failure as two clean outcomes.

Why this matters in interviews

"Why would you stream an AI response instead of waiting for the full completion?" tests whether a candidate reaches for the actual UX mechanism (perceived responsiveness under identical total latency) instead of the wrong-but-common answer that streaming makes the model itself faster.