Chunking Strategies

The plain-language idea

Before a document goes into a vector store, it gets split into smaller pieces called chunks. Each chunk gets its own embedding and is retrieved independently. This isn't an implementation detail you could skip — it's necessary for two concrete reasons:

• A whole document's embedding is too coarse. Averaging the meaning of an entire long document into a single vector blurs together everything it covers — a query about one specific paragraph won't match well against a vector representing the whole document's vague, blended topic.
• It won't fit in the model's context window anyway. Even if a single embedding could represent a long document well, retrieving and injecting the entire thing into a prompt burns far more context budget than the relevant portion actually needs.

The real design decision is chunk size, and it's a genuine tradeoff, not a "bigger is safer" default:

• Too small, and a chunk loses the surrounding context it needs to make sense on its own — retrieval returns fragments that are technically relevant but hard for the model to actually use.
• Too large, and the chunk's embedding gets diluted — it represents a blend of several ideas instead of one precise one, hurting retrieval accuracy, and it wastes context budget on text that isn't actually relevant to the query.

See it in a real thing: splitting a document with overlap

// Rough sketch of chunking with overlap
function chunkText(text: string, chunkSize: number, overlap: number) {
  const chunks: string[] = [];
  let start = 0;
  while (start < text.length) {
    chunks.push(text.slice(start, start + chunkSize));
    start += chunkSize - overlap; // step forward less than a full chunk — that's the overlap
  }
  return chunks;
}

A document, and one sentence that happens to fall right where a chunk boundary would land.

BackNext

Chunk overlap — where each chunk shares a bit of text with its neighbor, rather than chunks butting up cleanly end-to-end — exists to solve a specific failure mode: a sentence that happens to fall right at a chunk boundary otherwise gets awkwardly split in half, with neither resulting chunk containing the full sentence's meaning. Overlap makes it far more likely that any given sentence appears whole in at least one chunk, even if its "natural" position straddles a boundary.

What people usually miss

Watch out

• Chunking isn't optional overhead — it's what makes retrieval precise at all. Skipping it and embedding whole documents doesn't just work worse, it defeats the point of chunk-level retrieval entirely.
• Bigger chunks aren't automatically safer. It feels intuitive that "more context per chunk can't hurt," but oversized chunks dilute the embedding's precision and burn context budget on irrelevant surrounding text.
• Overlap costs storage and embedding compute, on purpose. Overlapping regions get embedded and stored more than once — that redundancy is the price paid for not losing boundary-spanning meaning, not an accident to optimize away.
• Chunk boundaries that ignore document structure (splitting mid-sentence, mid-code-block, mid-table-row) produce worse retrieval than boundaries that respect it. Naive fixed-length splitting is a reasonable starting point, not the ceiling — splitting on natural boundaries (paragraphs, sections, code blocks) usually retrieves better.

Why this matters in interviews

"A RAG feature keeps retrieving irrelevant or incomplete-feeling context — what would you check?" is a real diagnostic question, and chunk size and overlap are two of the first, cheapest things to inspect before assuming the embedding model or the vector database is the problem.