Skip to content

Vision RAG

Every RAG technique on this site so far starts the same way: extract text from a document, then chunk it. That extraction step is itself a failure point — a table's numbers get flattened into a sequence that loses which value belongs to which row and column; a multi-column layout gets read in the wrong order; a chart's legend ends up disconnected from its data. Vision RAG skips extraction entirely: render each page as an image, retrieve over the images directly, and let a vision-capable LLM read the retrieved page the way a person would — by looking at it.

How this differs from Multimodal RAG

Multimodal RAG extracts individual figures out of a PDF and retrieves text chunks plus those figures — text is still extracted and chunked either way. Vision RAG never extracts text at all: a whole page is the retrieval unit, embedded directly as pixels, and the page image itself is what the LLM reads at generation time.

Hands-on

The full comparison below is runnable, against the same sample paper used across this site: Vision RAG → in the code repo. Retrieval is fully local; generation needs a vision-capable LLM key (Anthropic or OpenAI).

TL;DR — quick revision
  • No text extraction anywhere: each PDF page is rendered as an image, embedded directly, and — when retrieved — read by the LLM as an image, not OCR'd text
  • Late interaction (MaxSim), not a single vector: a page is embedded as a grid of patch vectors, not pooled into one summary vector, so a page can win retrieval on the strength of one region — a single table cell — even if the rest of the page is unrelated to the query
  • Not a universal upgrade over naive text RAG: measured Recall@3 on 8 plain factual questions, naive chunk-based text RAG actually scores higher (1.000 vs. 0.875) — page-level retrieval is a coarser unit than a precise chunk, and on a mostly plain-text document, chunking's precision has room to win
  • Where it genuinely wins: asked a question whose answer is a specific table cell, naive text RAG retrieves a passage that only mentions the table without containing its data, and the LLM correctly declines to guess; Vision RAG retrieves the table's page image and reads the correct value straight off it
  • The smallest usable model in the ColPali family still needs a real fix to load: its packaged checkpoint config points at a training-time-only local path; the practical fix is a documented, reusable one — not a workaround to avoid mentioning

How a page becomes patches

Before any embedding happens, a page image goes through a fixed preprocessing pipeline, and each stage's output feeds the next. Verified directly on a real page from this recipe's sample PDF, rendered at 1275×1650px:

Stage What happens The arithmetic Result
1. Tiling The page is cut into a grid of tiles, each resized to the model's fixed 512×512px input size. This page splits into a 4×4 grid (16 local tiles), plus 1 extra tile holding a downscaled thumbnail of the whole page — so the model retains some sense of global layout even though each of the other 16 tiles only sees a local crop. 16 local tiles + 1 thumbnail tile 17 tiles total
2. Patchifying one tile Each 512×512 tile is cut into non-overlapping 16×16px patches (a patch-embedding layer is a convolution with stride equal to its kernel size, so every pixel belongs to exactly one patch — never overlapping). How many patches fit along one edge? 512 ÷ 16 = 32 patches/edge A 32×32 grid = 1,024 raw patches, in one tile
3. Merging (pixel shuffle) Every non-overlapping 4×4 block of adjacent raw patches is squashed into a single token — the "4" here is the model's scale_factor setting, a real compute/quality trade-off (4× fewer tokens to run late interaction over, at the cost of coarser localization within a tile). How many 4-patch blocks fit along one edge of the 32×32 grid? 32 ÷ 4 = 8 blocks/edge An 8×8 grid = 64 merged tokens, per tile (down from 1,024)
4. Totaling across the page Every one of the 17 tiles goes through steps 2-3 independently, each producing its own 64 tokens. 17 tiles × 64 tokens/tile 1,088 image tokens for the whole page

The 1,088 figure is confirmed directly by counting the actual image-placeholder tokens the processor emits for this page — it isn't a fixed constant of the model, it falls out of the page's pixel dimensions relative to the fixed 512×512 tile size, so a differently-shaped or differently-rendered page image would tile into a different number of tiles and produce a different count.

What one merged token actually "sees": each of the 64 tokens per tile corresponds to a 4×4 block of the tile's 32×32 raw patches, i.e. roughly a 64×64-pixel region of that 512×512 tile — about an eighth of the tile's width and height. Mapped back to the original page crop that tile came from, that's enough area to cover a short table cell, a few words, or a small piece of a diagram, but not a whole paragraph or a whole table — which is exactly the granularity late interaction needs to let one strong local match stand out.

Scoring: late interaction (MaxSim), not a single vector

Once a page is a grid of tokens (1,088 of them here, each a 128-dim vector after the model's projection) and a query is a short sequence of token vectors (typically 15-25 for a plain-language question, also 128-dim), scoring sums the best match each query token finds anywhere on the page:

score(query, page) = Σ (for each query token) max (over all page patches) dot_product

Concretely, with a 3-token query and a 4-token page (real runs use ~20 and ~1,088, but the mechanism is identical at any size): compute every query-token-vs-page-token dot product, giving a 3×4 grid of similarity scores. For each of the 3 query tokens (each row of that grid), keep only the single highest value — its best match anywhere on the page. Sum those 3 per-token maxima to get the page's final score. A different page gets scored the same way, independently, and the page with the higher summed score ranks first.

This is the same mechanism ColBERT uses for text retrieval, with image patches standing in for document tokens. The practical consequence: a page can win because ONE region of it matches strongly — one table cell, one line of a chart's legend — even if the rest of the page has nothing to do with the query. A single pooled vector per page couldn't do this; averaging a strong local match together with everything else on the page would dilute it away.

The trade-off is index size: storing ~1,088 vectors per page is far heavier than one vector per text chunk. For a handful of pages this doesn't matter; at real document-collection scale it's the main practical cost of this approach, and production setups typically use a vector store with native multi-vector support (Qdrant, Vespa) plus compression rather than storing raw patch vectors.

The model, and a checkpoint requirement worth knowing about

vidore/ColSmolVLM-256M-base — the lightest model in the ColPali/ColVision family with a usable ViDoRe benchmark score (~80 NDCG@5, vs. ~89 for the 2-3B leaders like ColQwen2.5). It's the one ColVision checkpoint that's plausibly runnable on an 8GB machine without a dedicated GPU — the larger models in this family realistically need one.

This specific checkpoint's adapter_config.json sets base_model_name_or_path to ./models/ColSmolVLM-256M-Base, a local path from its own training run that doesn't exist anywhere else. transformers auto-detects the adapter config on load and tries to resolve that path as a Hugging Face repo id, which fails. The checkpoint's full merged weights are already present in its model.safetensors, so the practical fix is to download the snapshot, patch a local copy of adapter_config.json to point at the real base model (HuggingFaceTB/SmolVLM-256M-Base, listed on the model card), and load from that local copy. The code below does this automatically.

Rendering pages, and loading the model

Rendering is the entire "extraction" step this approach needs — no OCR, no text parsing:

def render_pdf_pages(pdf_path: str, dpi: int = 150) -> list:
    """Render every PDF page to a PIL image. This is the entire
    "extraction" step Vision RAG needs — no text, no OCR."""
    import fitz  # pymupdf
    from PIL import Image

    doc = fitz.open(pdf_path)
    pages = []
    for page in doc:
        pixmap = page.get_pixmap(dpi=dpi)
        img = Image.frombytes("RGB", [pixmap.width, pixmap.height], pixmap.samples)
        pages.append(img)
    return pages

def load_vision_model(repo_id: str = "vidore/ColSmolVLM-256M-base"):
    """Load a ColPali-family page-image retriever.

    This checkpoint's `adapter_config.json` ships with
    `base_model_name_or_path` set to a local training path
    (`./models/ColSmolVLM-256M-Base`) that doesn't exist outside the
    original training run. `transformers` auto-detects the adapter config
    and tries to resolve that path as a Hugging Face repo id, which fails.
    The full merged weights are already present in `model.safetensors`, so
    the fix is to patch a local copy of `adapter_config.json` to point at
    the real base model (`HuggingFaceTB/SmolVLM-256M-Base`, from the model
    card) and load from that local copy."""
    import json
    import shutil
    from pathlib import Path

    import torch
    from huggingface_hub import snapshot_download
    from colpali_engine.models import ColIdefics3, ColIdefics3Processor

    snapshot_dir = snapshot_download(repo_id)
    patched_dir = Path("/tmp/colsmolvlm_patched")
    if patched_dir.exists():
        shutil.rmtree(patched_dir)
    shutil.copytree(snapshot_dir, patched_dir)

    for fname in ["adapter_config.json", "adapter2_config.json"]:
        fpath = patched_dir / fname
        if fpath.exists():
            config = json.loads(fpath.read_text())
            config["base_model_name_or_path"] = "HuggingFaceTB/SmolVLM-256M-Base"
            fpath.write_text(json.dumps(config))

    device = "mps" if torch.backends.mps.is_available() else "cpu"
    model = ColIdefics3.from_pretrained(str(patched_dir), torch_dtype=torch.float32).to(device).eval()
    processor = ColIdefics3Processor.from_pretrained(str(patched_dir))
    return model, processor

Indexing and searching page images

Every page is embedded as a full patch grid, and search scores every page against the query with MaxSim:

def build_page_image_index(pages: list, model, processor) -> list:
    """Embed every page as a multi-vector — one embedding per image patch,
    ColBERT-style — not a single pooled vector. Late interaction scoring
    needs the full patch grid."""
    import torch

    device = next(model.parameters()).device
    embeddings = []
    with torch.no_grad():
        for page in pages:
            batch = processor.process_images([page]).to(device)
            page_embedding = model(**batch)
            embeddings.append(page_embedding[0].cpu())
    return embeddings


def vision_search(query: str, page_embeddings: list, model, processor, k: int = 3) -> list:
    """Score every page with MaxSim (late interaction): for each query
    token embedding, take the best-matching patch embedding on the page,
    then sum those per-token maxima. A page can win because ONE region of
    it — one table cell, one chart legend line — matches strongly, even if
    the rest of the page doesn't."""
    import torch

    device = next(model.parameters()).device
    with torch.no_grad():
        query_batch = processor.process_queries([query]).to(device)
        query_embedding = model(**query_batch)[0].cpu()

    scores = [
        processor.score_multi_vector(query_embedding.unsqueeze(0), page_embedding.unsqueeze(0)).item()
        for page_embedding in page_embeddings
    ]
    ranked = sorted(enumerate(scores), key=lambda pair: pair[1], reverse=True)
    return ranked[:k]

Compared directly against the naive text RAG baseline used across this site — extract text, chunk it, embed with a dedicated text embedder:

def naive_text_search(pdf_path: str, query: str, k: int = 3) -> list:
    """The baseline this page compares against: extract text, chunk it,
    embed chunks with a dedicated text embedder. Same pattern used
    throughout this repo."""
    from langchain_text_splitters import RecursiveCharacterTextSplitter
    from pypdf import PdfReader
    from sentence_transformers import SentenceTransformer, util

    reader = PdfReader(pdf_path)
    full_text = "\n\n".join(page.extract_text() or "" for page in reader.pages)
    splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
    chunks = splitter.split_text(full_text)

    model = SentenceTransformer("all-MiniLM-L6-v2")
    chunk_embeddings = model.encode(chunks, convert_to_numpy=True, show_progress_bar=False)
    query_embedding = model.encode(query, convert_to_numpy=True)
    sims = util.cos_sim(query_embedding, chunk_embeddings)[0]
    top_k = sims.topk(min(k, len(chunks)))
    return [(chunks[i], sims[i].item()) for i in top_k.indices.tolist()]

Measured: naive text RAG wins on plain factual Recall@3

On the same 8 factual questions used across this repo, against the same "Attention Is All You Need" PDF:

                                 Recall@3
Naive text RAG (chunk-level)    1.000
Vision RAG (page-level)         0.875

The one question Vision RAG misses: "How many attention heads did they use?" retrieves the paper's attention-visualization figure pages — which literally show per-head attention diagrams — ahead of the page with the actual answer (h = 8, in a results table). The query's wording matches those figures visually and semantically more strongly than it matches a small table cell. Naming the table directly ("...per Table 3?") retrieves the right page. This is a real, structural property of page-level retrieval on a mostly plain-text document: a page is a coarser unit than a precise 500-character chunk, so a chunker's precision has room to win outright when the document has little layout complexity to begin with.

Where Vision RAG actually wins: table-layout integrity

Asked "What BLEU score did the base model get on the newstest2013 development set, per Table 3?" (the correct value is a specific table cell: 25.8):

=== Naive text RAG ===
Top retrieved chunk: "...establishing a new state-of-the-art BLEU score
of 28.4. The configuration of this model is listed in the bottom line
of Table 3. Training took 3.5 days on 8 P100 GPUs..."

Answer: I cannot determine what BLEU score the base model achieved on
the newstest2013 development set. [...] The context only explicitly
states that their best model achieved a BLEU score of 28.4, but does
not provide the specific score for the base model.

=== Vision RAG ===
Top retrieved page: the full Table 3 page image

Answer: According to Table 3, the base model achieved a BLEU score of
25.8 on the newstest2013 development set.

Naive text RAG retrieves a passage that mentions Table 3 and a BLEU score in the same breath, without containing Table 3's actual data — 28.4 is a different model's headline test-set score from a separate table entirely. The LLM, working only from that chunk, correctly declines to guess rather than repeating the nearby number as if it answered the question. Vision RAG retrieves the actual table's page image and reads the real cell value directly off it. This is the failure mode Vision RAG exists to fix: text chunking can separate a table's meaning from its data long before retrieval even runs, and no amount of clever prompting recovers what the chunk boundary already destroyed.

When to use which approach

Use naive text RAG for: plain-text-heavy documents — narrative prose, prose-style reports, most academic papers' body text. Chunk-level retrieval is a finer-grained, cheaper unit, and this page's own measurement shows it can out-recall page-level retrieval when there's little layout complexity to gain from. Use Vision RAG for: documents where a value's meaning depends on its position — financial tables, forms, multi-column layouts, charts, infographics. The measured example above is exactly this case: a number is meaningless without knowing which row and column it came from, and only reading the actual layout gets that right. Consider both, routed by document type: a production system indexing a mixed corpus (plain reports alongside dense tables) doesn't have to pick one globally — this is the same modality-routing idea covered on the Modular RAG page, applied to text-vs-layout instead of retrieval strategy.

Scenario Check

Sources & further reading