Tokens & Tokenization

The plain-language idea

A token is the actual unit an LLM reads, writes, and gets billed in. It's not a character and it's not a word — it's a sub-word chunk produced by a tokenizer, a separate piece of software that breaks text apart before any of it reaches the model.

Common short words are often a single token. Longer or rarer words frequently split into two or three pieces — a word like "tokenization" might come apart as token + ization. Punctuation, whitespace, and even parts of numbers can each claim their own token. None of this maps cleanly onto "one word" or "one character," which is exactly why it surprises people.

Two things ride directly on token count:

• Cost — nearly every LLM API prices per token, input and output counted separately.
• Context window — the model's total token budget for a request (covered in the next lesson) is measured in tokens, not characters or words.

So when you're estimating how much a prompt will cost or whether a document will fit, "count the characters" or "count the words" is the wrong instrument. You need a token count, and the only reliable way to get one is to actually run the text through the same tokenizer the model uses.

See it in a real thing: estimating a prompt's cost before sending it

// Rough sketch of what a token-aware cost estimate needs
const systemPrompt = "You are a helpful assistant for a Next.js codebase.";
const userMessage = "Summarize the changes in lib/ai/chat/handler.ts";

// Wrong: treating characters as a proxy for tokens
const naiveEstimate = (systemPrompt + userMessage).length; // character count — not billable units

// Right: run the actual text through a tokenizer to get a real count
// const tokenCount = tokenizer.encode(systemPrompt + userMessage).length;

A plain sentence: "Tokenization isn't 1:1 with words." 6 words, 40 characters.

BackNext

If you built a "will this fit?" check using .length on a string, it would be wrong in both directions — sometimes overestimating (short common words), sometimes badly underestimating (a paragraph of rare technical terms, code identifiers, or non-English text, all of which tend to tokenize less efficiently than plain English prose). A codebase like this app's own lib/ai/chat/handler.ts, which sits in front of every AI call, is exactly the place this matters: before a request goes out, you want a real token count for the assembled prompt, not a guess based on string length.

What people usually miss

Watch out

• A token is not a word. Some words are one token; others split into several. There's no fixed ratio you can rely on across all text — "roughly 4 characters per token" is a rough average for English prose, not a rule.
• Tokenization happens before the model ever "sees" the text. The tokenizer is a separate, fixed vocabulary lookup — the model doesn't choose how to split text, it only ever operates on whatever tokens the tokenizer already produced.
• Different models can use different tokenizers. A token count from one model's tokenizer isn't guaranteed to match another model's count for the same text — don't assume a number computed for one provider transfers to another.
• Output tokens are usually priced differently (often higher) than input tokens. A cost estimate that only counts the prompt and ignores the expected response length is measuring half the bill.
• Whitespace and punctuation count too. A long run of formatting, indentation, or repeated symbols consumes real tokens even though it "feels" free.

Why this matters in interviews

"How would you estimate the cost of a prompt before sending it?" is a real question that separates candidates who reach for a real tokenizer from ones who reason about character or word counts — and it opens naturally into context-window and cost-control follow-ups.