Caching & Prompt Caching

The plain-language idea

Every request to an LLM API sends the full prompt as tokens, and the provider has to process every one of those tokens before it can generate a response — that processing costs money and takes time, even for the part of the prompt that's identical to the last request. Prompt caching is a provider feature that lets you mark a prefix of your prompt (the part that doesn't change between calls) as cacheable, so on subsequent requests the provider can reuse its already-computed internal representation of that prefix instead of reprocessing it from scratch.

The part worth being precise about: it's a prefix. Caching works because the provider can recognize "the first N tokens of this request are byte-for-byte identical to the first N tokens of a request I processed recently." That means only content that's stable and comes first in the prompt is cacheable — a long, unchanging system prompt, or a large retrieved document that's the same across a session, both qualify. The user's actual message, which changes every turn and comes last, doesn't.

This isn't a niche optimization — for any feature that sends a large, mostly-static system prompt or context block on every call (which describes most production LLM features, including this app's own tutor and interview prompts), caching is a direct, meaningful lever on both cost (cached tokens are billed at a steep discount) and latency (skipping reprocessing means a faster response).

See it in a real thing: a tutor prompt sent on every message

System prompt (2,000 tokens, identical every single call):
  "You are a tutor. Answer only from the provided lesson context.
   Lesson context: <full lesson content>..."

User message (changes every turn):
  "Can you explain this differently?"

Without caching, every single turn in a tutoring conversation reprocesses that entire 2,000-token system prompt from zero, even though it hasn't changed since the conversation started. With prompt caching, the provider recognizes the identical prefix on the second, third, and every subsequent call in that session and skips reprocessing it — you're billed the discounted cached rate for those tokens instead of the full rate, on every turn after the first.

What people usually miss

Watch out

• Caching only helps stable prefixes, not the whole prompt. If your "system prompt" is actually reconstructed with slightly different content on every call (a timestamp, a random example, reordered context), it stops matching the cached prefix and you get no benefit — stability and ordering both matter.
• Content that changes every request has nothing to cache. If a feature's entire prompt is unique per call (e.g. summarizing a different document each time with no repeated instructions), caching has nothing to hold onto — this is the honest "when it doesn't help" case, not a caching failure.
• Put stable content first, variable content last. Because caching matches on a shared prefix, structuring a prompt as [stable system instructions] + [stable large context] + [variable user message] gets meaningfully more cache reuse than interleaving stable and variable content.
• Caching is a cost/latency optimization, not a correctness one. A cache hit and a cache miss must produce the same answer for the same input — caching should never change what the model is told, only how much reprocessing it costs to tell it.

Why this matters in interviews

"Our AI feature's costs are too high — what would you look at first?" is a real question, and a stable, repeated system prompt or RAG context that isn't using prompt caching is one of the first concrete, load-bearing answers worth naming.