Temperature & Sampling Parameters

The plain-language idea

At every step of generating a response, a model computes a probability distribution over what the next token could be — some tokens are far more likely than others given everything so far. Temperature controls how that distribution gets used to actually pick a token.

• Low temperature sharpens the distribution toward the most likely token — the model reliably picks what it considers the best next token, which reads as deterministic and consistent. Taken to the extreme (temperature at or near zero), output becomes highly repetitive and predictable.
• High temperature flattens the distribution, giving lower-probability tokens a real chance of being picked — output reads as more varied and creative, but also more likely to wander into something incoherent or off-topic.

Two related parameters constrain which tokens are even eligible to be sampled, independent of temperature:

• Top-p (nucleus sampling) — only sample from the smallest set of tokens whose cumulative probability adds up to p (e.g. the top 90% of probability mass), cutting off the long tail of unlikely tokens entirely.
• Top-k — only sample from the k most likely tokens, a simpler fixed-count cutoff.

These aren't mutually exclusive with temperature — an API call can set temperature and top-p and top-k together, each narrowing the pool or reshaping it in a different way.

See it in a real thing: picking parameters for two different features

// A structured-extraction endpoint: pulling a fixed JSON shape out of user input.
// Reliability matters far more than variety — low temperature.
const extractionParams = { temperature: 0.1 };

// A brainstorming endpoint: generating varied marketing headline ideas.
// Variety is the actual goal here — higher temperature is appropriate.
const brainstormParams = { temperature: 0.9, top_p: 0.95 };

The extraction endpoint benefits from the model reliably picking its most-confident token at every step, because the output needs to parse as valid, predictable JSON every time. The brainstorming endpoint benefits from the opposite: sampling more broadly is the entire point, since a handful of near-identical "safe" headlines would defeat the purpose of asking for options.

What people usually miss

Watch out

• Temperature doesn't change what the model "knows" — it changes how it samples from the probabilities it already computed. It's a decoding-time control, not a knowledge or capability control.
• Low temperature isn't automatically "more correct." It makes output more consistent and predictable, not more factually accurate — a confidently wrong answer at low temperature is still wrong, just reliably so.
• Temperature and top-p/top-k solve different problems and can be combined. Temperature reshapes the distribution's sharpness; top-p/top-k restrict the eligible pool. Using them together is normal, not redundant.
• Temperature 0 is not a hard reproducibility guarantee. Treat it as "very likely consistent," not "guaranteed byte-identical," especially across different infrastructure or model versions.
• Cranking temperature up doesn't fix a genuinely wrong or missing capability. If a model can't do a task correctly at low temperature, higher temperature usually just adds noise rather than unlocking correctness.

Why this matters in interviews

"You're building a feature that extracts structured data versus one that brainstorms ideas — how would you configure sampling differently for each?" tests whether a candidate understands temperature as a real engineering lever tied to the task's actual requirements, not a vague "creativity slider."