Context Windows

The plain-language idea

The context window is the maximum number of tokens a model can process in a single request, counting everything: the system prompt, the entire conversation history you send back with each turn, any documents or retrieved snippets you've stuffed in for grounding, the user's new message, and the model's own response. It's one pool, not separate allowances for each piece.

That last part is the part people miss. There's no dedicated "history budget" and "response budget" — every category is drawing from the same total. A long conversation history leaves less room for the response. A large retrieved document leaves less room for both.

When a request would exceed the window, one of two things happens, and which one depends entirely on how the calling application is built, not on the model itself:

• Truncation — the application (not the model) drops older content, usually the earliest turns of the conversation, to make everything fit.
• A hard error — the API rejects the request outright for exceeding the limit, and nothing gets processed at all.

The model itself doesn't "decide" to truncate anything — it never sees the token count. That logic lives in whatever code assembles the request before it goes out, which means it's the application developer's responsibility to manage, not something the API handles for free.

See it in a real thing: a chat handler assembling a request

// A simplified view of what lib/ai/chat/handler.ts has to reason about
// before every request goes out
const request = {
  systemPrompt,        // persistent instructions — fixed cost, every turn
  history: messages,   // grows with every back-and-forth in the conversation
  retrievedContext,     // documents pulled in for grounding, if any
  newMessage,           // the user's latest turn
};
// all four of the above, PLUS the model's response, share one token budget

A fixed-size window: the model's total token budget for this one request.

BackNext

A chat feature that just appends every past message to history forever will eventually hit this wall. The fix isn't "use a model with a bigger window" as a default reflex — it's deciding, deliberately, what actually needs to stay in context: maybe a running summary of older turns instead of the full transcript, maybe only the most relevant retrieved chunks instead of every document that matched a search.

What people usually miss

Watch out

• The context window is one shared budget, not separate slots. System prompt, history, retrieved documents, the new message, and the response all draw from the same total token count.
• The model doesn't truncate itself — the application does. Whether old messages get dropped, a document gets trimmed, or the request just fails with an error is a decision made in application code, not model behavior.
• A bigger window isn't automatically better. More tokens processed means more cost, and research has repeatedly shown models can struggle to find and use information placed in the middle of a very long context, even when it technically fits.
• "Fits in the window" and "the model will actually use it well" are different claims. Cramming in every possibly-relevant document is not the same as giving the model a well-curated, focused context.
• Streaming a response doesn't change the token budget. Streaming affects how the response arrives (incrementally vs. all at once), not how many tokens the request is allowed to consume.

Why this matters in interviews

"A user's conversation is getting long and responses start failing — what's happening and how do you fix it?" is a direct test of whether a candidate understands the context window as one shared budget, versus assuming the platform just handles unlimited history for free.