Skip to content

Build a Multi-Document Production RAG Pipeline

Every other recipe on this site tests one technique against one small, single-document corpus, on purpose — isolating a variable is the right way to measure what a specific technique actually does. This tutorial does the opposite: it combines chunking-strategy choice, retrieval-method choice, a real vector database, and RAGAS evaluation into one pipeline, run against three real papers at once (not one), so the comparisons and failure modes below are the ones that only show up once several pieces are stacked together.

The code lives in rag-cookbook/multi-doc-production-rag/; every block below is pulled live from those exact files at build time.

Read these first

This tutorial assumes you've read Chunking Strategies, Retrieval Methods, Vector Databases, and RAGAS Deep Dive — it reuses their techniques directly and doesn't re-explain how each one works internally.

The corpus

Three real, closely-related papers — "Attention Is All You Need," "BERT," and "GPT-3" — roughly 230 pages combined, producing 700 to 1,000+ chunks depending on strategy. Being closely related (all about language model pretraining and architecture) makes cross-document confusion a real, testable risk: a question like "how many layers does the model have" is genuinely ambiguous across all three, each with a real, different, correct answer.

A 9-question eval set spans all three documents, with a real keyword from that paper's actual text backing each question — the same keyword-derived ground truth pattern used on the Evaluation Metrics page, extended here to also check that a retrieved chunk came from the right document, not just that it contains the right keyword.

1. Install

git clone https://github.com/DhruvMakwana/rag-cookbook.git
cd rag-cookbook/multi-doc-production-rag
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

sentence-transformers>=6.0
pypdf>=6.0
requests>=2.31
numpy>=2.0
langchain-text-splitters>=0.2
rank_bm25>=0.2
python-dotenv>=1.0

# Vector database
qdrant-client>=1.19

# RAGAS evaluation (needs langchain-community pinned -- see README)
langchain-community==0.3.31
ragas>=0.4.3
langchain-anthropic>=0.3
langchain-huggingface>=0.1
anthropic>=0.34
openai>=1.0
pandas>=2.0

2. Configure

cp .env.example .env

The chunking and retrieval comparisons need no API key at all. The final RAGAS evaluation step needs an LLM — defaults to a local Ollama model (ollama serve, ollama pull qwen3:4b), or set LLM_PROVIDER=anthropic/openai in .env.

3. Chunking strategies, compared on the same multi-document corpus

def chunk_fixed_size(pages: list, size: int = 400, overlap: int = 50) -> list:
    """`pages` is a list of (doc_name, page_number, page_text) tuples --
    chunking per page (rather than guessing offsets in a merged string)
    makes page attribution exact by construction."""
    chunks = []
    chunk_id = 0
    for doc_name, page_num, page_text in pages:
        for start in range(0, len(page_text), size - overlap):
            piece = page_text[start:start + size].strip()
            if len(piece) < 40:
                continue
            chunks.append({"id": chunk_id, "text": piece, "doc": doc_name, "page": page_num})
            chunk_id += 1
    return chunks


def chunk_recursive(pages: list, size: int = 400, overlap: int = 50) -> list:
    from langchain_text_splitters import RecursiveCharacterTextSplitter

    splitter = RecursiveCharacterTextSplitter(chunk_size=size, chunk_overlap=overlap, separators=["\n\n", "\n", ". ", " ", ""])
    chunks = []
    chunk_id = 0
    for doc_name, page_num, page_text in pages:
        for piece in splitter.split_text(page_text):
            if len(piece.strip()) < 40:
                continue
            chunks.append({"id": chunk_id, "text": piece.strip(), "doc": doc_name, "page": page_num})
            chunk_id += 1
    return chunks


def chunk_sentence_window(pages: list, sentences_per_chunk: int = 5, overlap_sentences: int = 1) -> list:
    """Groups N sentences per chunk with a 1-sentence overlap, instead
    of splitting mid-sentence at a fixed character count -- a cheap
    proxy for "semantic" chunking with no embedding-based splitter."""
    import re

    chunks = []
    chunk_id = 0
    step = max(1, sentences_per_chunk - overlap_sentences)
    for doc_name, page_num, page_text in pages:
        sentences = re.split(r"(?<=[.!?])\s+", page_text)
        for start in range(0, len(sentences), step):
            piece = " ".join(sentences[start:start + sentences_per_chunk]).strip()
            if len(piece) < 40:
                continue
            chunks.append({"id": chunk_id, "text": piece, "doc": doc_name, "page": page_num})
            chunk_id += 1
    return chunks

def evaluate_chunking_strategy(chunks: list, eval_set: list, model, k: int = 3) -> dict:
    """`eval_set` items need question/keyword/doc/reference keys.
    Ground truth for a question is every chunk from the RIGHT document
    that contains the keyword -- catching wrong-document attribution as
    a real failure mode, not just wrong-chunk-within-the-right-doc."""
    import numpy as np

    texts = [c["text"] for c in chunks]
    embeddings = model.encode(texts, convert_to_numpy=True, normalize_embeddings=True, show_progress_bar=False)

    recalls, hits = [], []
    for item in eval_set:
        relevant = {c["id"] for c in chunks if item["keyword"].lower() in c["text"].lower() and c["doc"] == item["doc"]}
        query_vec = model.encode(item["question"], convert_to_numpy=True, normalize_embeddings=True)
        sims = embeddings @ query_vec
        retrieved = [chunks[i]["id"] for i in np.argsort(-sims)[:k]]

        hits_in_k = sum(1 for cid in retrieved if cid in relevant)
        recalls.append(hits_in_k / len(relevant) if relevant else 0.0)
        hits.append(1.0 if hits_in_k > 0 else 0.0)

    return {"num_chunks": len(chunks), "recall_at_k": float(np.mean(recalls)), "hit_rate_at_k": float(np.mean(hits))}

Measured Recall@3 / Hit Rate@3 across the 9-question eval set:

fixed      chunks=1011  recall@3=0.139  hit_rate@3=0.222
recursive  chunks=1017  recall@3=0.181  hit_rate@3=0.444
sentence   chunks= 743  recall@3=0.333  hit_rate@3=0.667

Sentence-window chunking — grouping complete sentences instead of splitting at a fixed character count — clearly outperforms both fixed-size and recursive chunking here, roughly tripling hit rate over fixed-size despite producing fewer, not more, chunks. On a corpus with dense technical claims packed into single sentences (parameter counts, layer counts, specific scores), keeping each sentence whole mattered more than chunk-size tuning.

4. Dense-only vs. hybrid retrieval, on the winning chunking strategy

def hybrid_rrf_fusion(bm25_ranks: list, dense_ranks: list, k_const: int = 60) -> list:
    """Reciprocal Rank Fusion: each ranker contributes 1/(k+rank) per
    item; scores from both rankers are summed. No mechanism exists to
    detect a ranker that's contributing pure noise for a given query --
    RRF trusts both rankers equally by construction."""
    scores = {}
    for rank, cid in enumerate(bm25_ranks, start=1):
        scores[cid] = scores.get(cid, 0.0) + 1.0 / (k_const + rank)
    for rank, cid in enumerate(dense_ranks, start=1):
        scores[cid] = scores.get(cid, 0.0) + 1.0 / (k_const + rank)
    return [cid for cid, _ in sorted(scores.items(), key=lambda x: -x[1])]

Measured against the same 9 questions, sentence-window chunks:

dense-only  recall@3=0.333  hit_rate@3=0.667
hybrid+RRF  recall@3=0.159  hit_rate@3=0.444

Hybrid search made retrieval worse here, and that's a real result

This isn't the usual outcome — Embedding Model Selection has a real case where hybrid RRF fusion clearly helps. Checking why hybrid underperforms on this eval set: none of the 9 questions have exact-match value (rare terms, product codes, IDs) that BM25 is actually good at finding. For a natural-language question like "how many attention heads does the base Transformer model use," BM25's own top-ranked result often isn't the chunk that actually answers it — its top hit for that exact question came from a different page entirely, one with high keyword overlap but the wrong content. RRF has no mechanism to detect that BM25 is contributing noise for a specific query; it sums both rankers' reciprocal-rank scores unconditionally, so a bad BM25 ranking can pull a correct dense-retrieval top-1 result out of the fused top-3. The lesson isn't "don't use hybrid search" — it's that hybrid search's benefit is conditional on the corpus and query style actually having exact-match value to contribute, which this particular eval set doesn't.

5. Indexing into a real vector database, with metadata filtering across documents

def index_and_query_qdrant(chunks: list, embeddings, query_vec: list, filter_doc: str = None, limit: int = 3):
    """Indexes a multi-document corpus into Qdrant with each chunk's
    source document as metadata, then queries with an optional
    document filter -- useful when a query is genuinely ambiguous
    across documents in the corpus (e.g. "how many layers does the
    model have" could be about any of several papers)."""
    from qdrant_client import QdrantClient, models

    client = QdrantClient(":memory:")
    dim = embeddings.shape[1]
    client.create_collection(
        collection_name="corpus",
        vectors_config=models.VectorParams(size=dim, distance=models.Distance.COSINE, hnsw_config=models.HnswConfigDiff(m=32, ef_construct=200)),
    )
    client.upsert(
        collection_name="corpus",
        points=[
            models.PointStruct(id=c["id"], vector=embeddings[i].tolist(), payload={"doc": c["doc"], "page": c["page"]})
            for i, c in enumerate(chunks)
        ],
    )

    query_filter = None
    if filter_doc:
        query_filter = models.Filter(must=[models.FieldCondition(key="doc", match=models.MatchValue(value=filter_doc))])

    result = client.query_points(collection_name="corpus", query=query_vec, limit=limit, query_filter=query_filter)
    return [(p.payload["doc"], p.payload["page"], p.score) for p in result.points]

Measured against the genuinely ambiguous question "How many layers does the model have?":

unfiltered:            attention_is_all_you_need.pdf p3 (0.444), attention_is_all_you_need.pdf p3 (0.420), gpt3.pdf p20 (0.380)
filtered to bert.pdf:  bert.pdf p3 (0.337), bert.pdf p5 (0.309), bert.pdf p8 (0.288)

Unfiltered, BERT's own answer (12 layers) never makes the top-3 at all — the Transformer and GPT-3 papers' own layer-count mentions score higher, since "how many layers" matches their phrasing at least as well. This is exactly the scenario metadata filtering exists for: when the calling application knows which document a question is actually about (a user viewing a specific paper, a chatbot scoped to one product's docs), a doc= filter turns an ambiguous multi-document retrieval into an unambiguous one — not a hypothetical use case, a real fix for a real miss measured above.

6. RAGAS evaluation of the assembled pipeline

Running naive RAG (sentence chunking, dense retrieval, top-3) over the full 9-question eval set and scoring with RAGAS's core four metrics:

faithfulness:                          0.8889
answer_relevancy:                      0.6067
llm_context_precision_with_reference:  0.6111
context_recall:                        0.6667

Per-question, 3 of 9 questions show context_recall = 0.0 — a real retrieval miss, not a generation problem: two of them (the Transformer's optimizer, GPT-3's context window) got an honest "the context doesn't specify" from the model rather than a hallucinated guess, which is the correct behavior when retrieval genuinely fails to surface the answer.

The same grounding failure mode measured on RAGAS Deep Dive, reproduced here

The third miss is the interesting one: asked "What are BERT's two pretraining tasks called?", the naive RAG pipeline answered "BERT is pre-trained using masked language modeling (MLM) and next sentence prediction (NSP)" — completely correct. It still scored faithfulness = 0.0, context_precision = 0.0, context_recall = 0.0. The retrieved top-3 chunks for that specific question didn't actually contain "Masked LM" or "NSP" — the model answered from its own training knowledge about a well-known paper, not from what retrieval surfaced. This is the identical failure mode RAGAS Deep Dive measured on a single-document corpus, reproducing here on a genuinely different, multi-document one — real evidence that the pattern isn't specific to one paper or one eval set.

Run it yourself

python multi_doc_rag.py --part chunking
python multi_doc_rag.py --part retrieval
python multi_doc_rag.py --part qdrant
python multi_doc_rag.py --part ragas
python multi_doc_rag.py --part all

Where this still breaks

  • The chunking and retrieval winners here are corpus-specific, not universal. Sentence-window chunking won on this eval set because its questions target single-sentence factual claims — a corpus with longer, multi-sentence arguments to retrieve might favor a different strategy entirely. Re-run the comparison on your own corpus before assuming these results transfer.
  • The eval set is small (9 questions). Real production systems need eval sets in the dozens-to-hundreds range before trusting a metric difference as more than noise — this tutorial's numbers are illustrative of the method, not a claim that sentence-window chunking universally beats recursive chunking.
  • This pipeline has no reranking, no query rewriting, and no retrieval-quality gate before generation. See Advanced RAG — Post-retrieval and Corrective RAG for the pieces this capstone deliberately leaves out to keep the comparison focused.

Sources & further reading