Chunking Strategies for RAG: Why Naive Splitting Breaks Retrieval

Chunking Strategies for RAG: Why Naive Splitting Breaks Retrieval

In retrieval-augmented generation, chunking decides whether the system retrieves a complete, useful fact or an incomplete fragment. Naive fixed-size splitting breaks tables, procedures and cross-references in supply-chain documents, so an assistant returns confidently wrong or unusable answers. Recursive splitters keep larger units such as paragraphs before falling back to sentences and words, and overlap reduces loss when context crosses a boundary. Neither alone guarantees that the facts a question needs end up in the same chunk.

The problem

A manufacturer builds a RAG assistant over procurement policies, supplier contracts, inventory SOPs, shipping instructions and quality documents. A planner asks: "What is the reorder point and safety stock for SKU A123 when supplier lead time is 21 days?" The answer spans a policy paragraph, an inventory table and a supplier appendix. If ingestion cuts every document into 500-character pieces, the SKU, the lead time and the formula can land in three different chunks. Retrieval then fails, not because the embedding model is poor, but because no chunk holds enough context. Good chunks are semantically complete, retrievable, traceable and small enough for the model's context window.

Naive chunking produces three typical failures. Context loss: "use a 95% service level" is separated from the formula it governs. Ambiguous retrieval: a chunk holding only "SKU A123" looks like every other SKU chunk. Broken structure: tables, multi-step procedures and contract clauses are cut mid-row or mid-clause.

1. Naive versus recursive splitting

All examples share one sample policy, which includes a numbered procedure inside its Exceptions section, as real SOPs do.

# sample_doc.py -- the policy used by every example
SOP = """Inventory Replenishment Policy
1. Purpose
This policy defines how replenishment quantities are calculated for finished goods and critical components.
2. Reorder Point
The reorder point is calculated as:
Reorder Point = Average Daily Demand x Lead Time + Safety Stock.
For SKU A123, average daily demand is 120 units and supplier lead time is 21 days.
3. Safety Stock
Safety stock is calculated as:
Safety Stock = Z x Standard Deviation of Daily Demand x sqrt(Lead Time).
For SKU A123, use a 95% service level, corresponding to Z = 1.65.
The standard deviation of daily demand is 25 units.
4. Exceptions
For supplier disruptions longer than 10 days:
1. Notify the category manager within 24 hours.
2. Escalate to the Supply Chain Planning Council.
"""

LangChain's RecursiveCharacterTextSplitter tries larger separators first (its defaults are paragraph break, line break, space, then single characters) and only falls back when a piece is still too large; LlamaIndex's SentenceSplitter likewise respects sentence boundaries. The script checks rather than eyeballs the outcome: does any single chunk contain every fact a safety-stock question needs?

# naive_vs_recursive.py -- pip install langchain-text-splitters
from langchain_text_splitters import CharacterTextSplitter, RecursiveCharacterTextSplitter
from sample_doc import SOP
# 1. Naive: cut every 220 characters regardless of meaning.
naive = CharacterTextSplitter(separator="", chunk_size=220, chunk_overlap=0).split_text(SOP)
# 2. Recursive: try paragraph, line, sentence, word boundaries in that order.
#    Sizes here are characters; embedding limits are tokens, so production code should
#    measure length with the embedding model's tokenizer (e.g. from_tiktoken_encoder).
recursive = RecursiveCharacterTextSplitter(
    chunk_size=350, chunk_overlap=50, separators=["\n\n", "\n", ". ", " ", ""]
).split_text(SOP)
def complete(chunks: list[str], *facts: str) -> bool:
    """Does any single chunk contain every fact the question needs?"""
    return any(all(f in c for f in facts) for c in chunks)
if __name__ == "__main__":
    need = ("Z = 1.65", "Standard Deviation of Daily Demand x sqrt(Lead Time)", "25 units")
    for name, chunks in (("naive", naive), ("recursive", recursive)):
        print(f"{name}: {len(chunks)} chunks, safety-stock inputs together: {complete(chunks, *need)}")

Running it shows the 220-character splitter cutting the safety-stock formula mid-word ("Standard Deviation o | f Daily Demand"), while the recursive splitter keeps the whole Safety Stock section together. It also shows the limit of recursive splitting: the reorder-point question needs A123's 21-day lead time from section 2 and the safety-stock inputs from section 3, and no single chunk holds both. That is a retrieval problem, solved by retrieving several sections or by parent-child retrieval, not by a bigger chunk size. Sizes here are characters; embedding limits are tokens, so production splitters should measure length with the model's tokenizer.

2. Section-aware chunking with context and provenance

For operational documents, a useful chunk is a coherent section plus metadata. A heading pattern that accepts any line starting with a number treats "1. Notify the category manager within 24 hours." as a new section, so it cuts the very procedure the article warns about. Here a heading must be a short numbered line that is not a sentence, and numbered steps stay inside their section. Every chunk, including pieces of an oversized section, is prefixed with the document title and heading. Each also carries document ID, version, part index and a content hash, so it can be cited, versioned and deleted. Unnumbered headings, PDF layout, scanned pages and page numbers need a real document parser; this is a demonstration of the policy, not a parser.

# section_chunker.py -- heading-aware chunks that carry their context and provenance
from __future__ import annotations
import hashlib
import re
from langchain_text_splitters import RecursiveCharacterTextSplitter
# A heading is a short numbered line ("3. Safety Stock", "3.1 Inputs") that is not a
# sentence. Numbered procedure steps end with punctuation and must stay in their section.
HEADING = re.compile(r"^(\d+(?:\.\d+)*)\.?\s+([^\n]{1,60})$")
def is_heading(line: str) -> bool:
    m = HEADING.match(line.strip())
    return bool(m) and not m.group(2).rstrip().endswith((".", ":", ";", ","))
def chunk_by_sections(doc: str, doc_id: str, version: str, max_chars: int = 600) -> list[dict]:
    lines = doc.strip().splitlines()
    title = lines[0].strip()                    # first line: document title
    sections, heading, buf = [], title, []
    for line in lines[1:]:
        if is_heading(line):
            if any(b.strip() for b in buf):
                sections.append((heading, "\n".join(buf).strip()))
            heading, buf = line.strip(), []
        else:
            buf.append(line)
    if any(b.strip() for b in buf):
        sections.append((heading, "\n".join(buf).strip()))
    splitter = RecursiveCharacterTextSplitter(chunk_size=max_chars, chunk_overlap=80,
                                              separators=["\n\n", "\n", ". ", " ", ""])
    chunks = []
    for heading, body in sections:
        parts = [body] if len(body) <= max_chars else splitter.split_text(body)
        for i, part in enumerate(parts):
            # Every piece repeats its title and heading, so a split-off fragment such as
            # "use Z = 1.65" still says which policy and section it belongs to.
            text = f"{title} > {heading}\n{part}"
            chunks.append({
                "chunk_id": f"{doc_id}:{version}:{len(chunks)}", "text": text,
                "metadata": {"doc_id": doc_id, "version": version, "title": title,
                             "heading": heading, "part": i, "parts": len(parts),
                             "sha256": hashlib.sha256(text.encode()).hexdigest()}})
    return chunks

Worked example. With section-aware chunks, "What safety stock should we use for SKU A123?" retrieves the Safety Stock section with its formula and inputs (95% service level, Z = 1.65, demand standard deviation 25 units). Safety stock = 1.65 × 25 × √21 ≈ 189.03, which rounds up to 190 units, giving a reorder point of 120 × 21 + 190 = 2,710 units (2,709 if safety stock is rounded to the nearest unit instead). The formula assumes constant lead time and independent, stable daily demand. The assistant should cite the section and validate units and assumptions.

3. Table-aware chunking

Splitting a table by characters separates rows from their headers and leaves values meaningless. Keep a small table whole, or turn each row into a self-contained record that repeats the column meanings and units. The SKU also belongs in metadata, not just the text, so retrieval can filter on it exactly instead of hoping similarity separates A123 from nearly identical rows. The function rejects tables with missing columns or duplicate SKUs, which would produce conflicting records. It uses itertuples, which keeps column types, rather than iterrows, which casts each row to one dtype and can turn integers into floats on all-numeric tables.

# table_chunks.py -- one self-contained, filterable record per table row
from __future__ import annotations
import pandas as pd
INVENTORY = pd.DataFrame([
    {"sku": "A123", "avg_daily_demand": 120, "lead_time_days": 21, "service_level": 0.95, "z": 1.65, "demand_std_dev": 25},
    {"sku": "B456", "avg_daily_demand": 45, "lead_time_days": 14, "service_level": 0.90, "z": 1.28, "demand_std_dev": 10},
    {"sku": "C789", "avg_daily_demand": 300, "lead_time_days": 30, "service_level": 0.98, "z": 2.05, "demand_std_dev": 60},
])
REQUIRED = {"sku", "avg_daily_demand", "lead_time_days", "service_level", "z", "demand_std_dev"}
def table_rows_to_chunks(df: pd.DataFrame, source_doc: str, version: str,
                         effective_date: str) -> list[dict]:
    if missing := REQUIRED - set(df.columns):
        raise ValueError(f"table is missing columns: {sorted(missing)}")
    if df["sku"].duplicated().any():
        raise ValueError("duplicate SKU rows would produce conflicting chunks")
    chunks = []
    for r in df.itertuples(index=False):        # itertuples keeps per-column dtypes
        text = (f"Inventory replenishment parameters for SKU {r.sku}: average daily demand "
                f"{r.avg_daily_demand} units/day; supplier lead time {r.lead_time_days} days; "
                f"service level {r.service_level:.0%} (Z = {r.z}); standard deviation of daily "
                f"demand {r.demand_std_dev} units. Source: {source_doc} v{version}, "
                f"effective {effective_date}.")
        chunks.append({
            "chunk_id": f"{source_doc}:{version}:sku:{r.sku}", "text": text,
            # The SKU lives in metadata too, so retrieval can *filter* on it instead of
            # hoping similarity separates A123 from near-identical rows.
            "metadata": {"record_type": "inventory_parameter", "sku": r.sku,
                         "source_document": source_doc, "version": version,
                         "effective_date": effective_date}})
    return chunks

Asked about "lead time and demand variability for SKU C789", a retriever filtering on sku = C789 returns exactly one record with its source SOP, version and effective date, removing the risk of mixing C789's values with A123's.

4. Parent-child retrieval

Small chunks retrieve precisely but carry little context; large ones carry context but retrieve poorly. Parent-child retrieval indexes small child chunks and gives the model their parent. The parent should be the logical section (the whole "Safety Stock" section), not a fixed-size window. Overlapping 900-character windows cut sections in arbitrary places and send duplicated text to the model. Children store a parent_id rather than a copy of the parent, and at query time ranked child hits are mapped to unique parents, so three children of one section do not fill the context with the same text three times.

# parent_child.py -- index small children, return whole sections (deduplicated) to the LLM
from __future__ import annotations
from langchain_text_splitters import RecursiveCharacterTextSplitter
from section_chunker import chunk_by_sections
def build_parent_child(doc: str, doc_id: str, version: str, child_size: int = 200,
                       child_overlap: int = 30) -> tuple[dict[str, dict], list[dict]]:
    """Parents are logical sections (not size-based windows, which overlap and cut the
    "Safety Stock" section in two). Children store a parent_id, not a copy of the parent."""
    parents = {c["chunk_id"]: c for c in chunk_by_sections(doc, doc_id, version, max_chars=1500)}
    splitter = RecursiveCharacterTextSplitter(chunk_size=child_size, chunk_overlap=child_overlap,
                                              separators=["\n", ". ", " ", ""])
    children = []
    for pid, parent in parents.items():
        heading = parent["metadata"]["heading"]
        body = parent["text"].split("\n", 1)[1]
        for i, piece in enumerate(splitter.split_text(body)):
            children.append({"child_id": f"{pid}:c{i}", "parent_id": pid,
                             "text": f"{heading}\n{piece}"})   # child keeps its heading
    return parents, children
def parents_for(hits: list[dict], parents: dict[str, dict], max_parents: int = 3) -> list[dict]:
    """Map ranked child hits to unique parents, keeping the rank of each parent's best child."""
    seen: dict[str, dict] = {}
    for hit in hits:
        if hit["parent_id"] not in seen:
            seen[hit["parent_id"]] = parents[hit["parent_id"]]
        if len(seen) == max_parents:
            break
    return list(seen.values())

Asked "What happens if a critical supplier has a disruption lasting more than 10 days?", the child holding "disruptions longer than 10 days" matches, and the model receives the full Exceptions section, both escalation steps included, with its source metadata for citation.

Common mistakes

One global chunk size for every document type, or character limits where the embedding model counts tokens.

Splitting tables, JSON, code or numbered procedures at arbitrary boundaries, or using a heading detector that mistakes steps for headings.

Zero overlap when sentences or requirements span boundaries, or so much overlap that near-duplicate chunks crowd results.

Chunks without title, heading and metadata ("use 1.65" with no context), and no versioning, so outdated policies outrank current ones.

Evaluating only generated answers instead of retrieval recall, precision and citation accuracy; assuming semantic chunking can fix poorly structured sources.

Production checklist

Profile documents first (SOPs, contracts, tables, email, PDFs, scans). Preserve structure by splitting on headings, clauses, list items and rows before applying limits, with recursive fallback inside oversized sections. Choose overlap deliberately, starting around 10–20% of chunk size and validating with retrieval metrics. Attach metadata: document ID, title, heading, page, effective date, version, region, product line, access level and, for tables, the entity key. Version every chunk with a content hash, ingestion time, source version and deletion status, and re-embed when the recipe changes. Evaluate retrieval: recall@k, precision@k, MRR, citation accuracy and faithfulness, on SKU-specific, multi-hop, comparative and time-sensitive questions. Monitor empty retrievals, low similarity, stale sources and duplicate chunks, and filter by permission so confidential supplier pricing is never retrieved by the wrong user.

Takeaway. Effective chunking is a data-modelling problem, not a text-processing setting. Preserve business meaning first, add traceable metadata second, and only then tune size and overlap, measuring the result with retrieval metrics.

Back to all blogs