Skip to content

Naive RAG

An LLM only knows what was baked into it at training time. Ask it about a document it's never seen — your company's refund policy, a contract you just signed, a paper published last week — and it either says so, or worse, answers confidently anyway from something that merely sounds related. Retrieval-Augmented Generation exists to fix this: instead of relying on what the model memorized, you hand it the relevant text at the moment it needs to answer, pulled fresh from a knowledge base you control.

Naive RAG is the simplest version of that idea — one retrieval pass, one generation pass, no loops, no quality checks, no second-guessing. It's not called "naive" as an insult; it's the baseline every other architecture on this site exists to improve on, and it's still the right choice for a surprising number of real use cases.

Hands-on

Prefer to build this instead of reading about it? Build Naive RAG From Scratch → — a runnable pipeline over a real PDF, pluggable across Anthropic, OpenAI, and local Ollama.

TL;DR — quick revision
  • Pipeline: chunk → embed → store (offline, once) · embed query → similarity search → top-k → prompt → generate (online, per query)
  • "Naive" = one retrieve-then-generate pass, no loops, no reranking, no self-checks
  • Five failure modes: semantic similarity ≠ relevance · no query understanding · fixed top-k with no reranking · no groundedness check · can't handle multi-part questions
  • Hard rule: query and chunks must be embedded with the exact same embedding model, or similarity scores become meaningless
  • Use it when: small, clean, well-scoped KB, low-stakes lookups, prototyping
  • Upgrade path: Advanced RAG (pre/post-retrieval) is the cheapest next step for almost every one of these failure modes

The core idea

Split your documents into chunks, turn each chunk into a vector that captures its meaning, and store those vectors. When a question comes in, turn the question into a vector too, and find the stored chunks whose vectors are closest to it — "closest" standing in for "most semantically similar." Stuff those chunks into the prompt, and let the LLM write an answer grounded in them instead of its own memory.

flowchart LR
    subgraph Offline["Indexing — done once"]
        A[Documents] --> B[Chunk]
        B --> C[Embed]
        C --> D[(Vector DB)]
    end
    subgraph Online["Per query"]
        E[User question] --> F[Embed query]
        F --> G[Similarity search]
        D --> G
        G --> H[Top-k chunks]
        H --> I[Prompt template]
        I --> J[LLM]
        J --> K[Answer]
    end

Two phases, run at very different times: indexing happens offline, once per document (or whenever the source changes); retrieval + generation happens online, once per user question, in milliseconds to a couple seconds.

Walking through it

Say your knowledge base is a company handbook, and a chunk of it reads:

"Employees may work remotely up to 3 days per week with manager approval. International remote work requires HR approval and is limited to 30 days per year."

1. Indexing. This chunk (along with every other chunk from every other document) gets run through an embedding model, producing a vector — a few hundred to a few thousand numbers that encode its meaning. That vector, plus the original chunk text and some metadata (source document, page number), gets stored in a vector database like FAISS or Milvus.

2. Retrieval. A user asks: "How many days can I work internationally remote?" That question gets embedded with the same embedding model used for indexing, and the vector DB returns the chunks whose vectors are nearest to it — usually by cosine similarity. The handbook chunk above should come back near the top, since "international remote work" and "internationally remote" land close together in vector space even though the wording differs.

3. Augmentation. The top-k retrieved chunks get dropped into a prompt template:

Answer the question using only the context below.
Context: {retrieved_chunks}
Question: {user_query}

4. Generation. The LLM reads that prompt and writes an answer — ideally "up to 30 days per year, with HR approval" — grounded in the retrieved text rather than guessed from its training data.

That's the whole pipeline. No loop, no verification step, no re-ranking — whatever came back from similarity search is what the LLM sees, for better or worse.

Where it breaks

The "for worse" part is the point of this page. Every one of these is a real failure mode, not a hypothetical:

  • Semantic similarity isn't the same as relevance. A chunk can be embedding-close to the query and still not contain the answer. A query about "refund policy for damaged items" can retrieve a chunk about "return policy for wrong-size items" — same neighborhood in vector space, wrong answer in practice.
  • There's no query understanding. A vague or oddly-phrased question, or one that uses different vocabulary than the source documents, can cause the similarity search to miss the right chunk entirely — the system has no way to notice the phrasing was the problem.
  • Top-k is fixed and unranked. Whatever the nearest-neighbor search returns is used as-is. There's no second pass asking "which of these k chunks is actually most relevant" — noisy, tangential chunks dilute the context right alongside the good ones.
  • Nothing checks groundedness. Nothing stops the LLM from hallucinating even with good context in front of it, and nothing stops it from confidently answering when the retrieved context has nothing useful — it should say "I don't know" here, but naive RAG never taught it to.
  • Multi-part questions don't work well. A single query embedding represents one point in vector space. A question like "compare X's pricing and Y's pricing" is really two lookups wearing a trenchcoat, and one embedding can't represent both halves at once.

Framing for interviews

Every architecture covered elsewhere on this site — Advanced RAG, Modular RAG, Self-RAG, CRAG, Agentic RAG — exists to patch one or more of these exact five failure modes. None of it is sophistication for its own sake; each addition maps to a specific way naive RAG breaks.

When naive RAG is actually the right call

Despite the list above, don't reflexively upgrade. Naive RAG is the correct choice for simple, well-scoped Q&A over a small, clean knowledge base — a FAQ bot, an internal wiki lookup, a prototype you're validating before investing further. The moment you're in production with real users, real-world phrasing, and any cost to being wrong, move at minimum to Advanced RAG's pre- and post-retrieval techniques — they're the cheapest, highest-leverage upgrades from here.

Scenario Check

Six scenario-based questions to test whether the failure modes above actually stuck — the kind of "what would happen if..." framing that tends to come up in system-design interviews. Nothing here is stored or scored beyond this page view; it's just a self-check.