Skip to content

Multimodal RAG

Every technique so far retrieves text. But a real document rarely is only text — the same paper this whole site uses as a running example has a genuine architecture diagram, an attention-mechanism diagram, a multi-head attention diagram, sitting right alongside the prose. Multimodal RAG is retrieval across genuinely different modalities — text, images, audio, video, or any combination — recognizing that the right answer to a question sometimes isn't a paragraph, it's a picture.

Scope of this page

Multimodal RAG covers any modality mix — text+audio, text+video, all of the above at once. This page (and its code) scopes down to just text + images, using CLIP-family models, since that's the combination with the most established, testable tooling and the clearest real corpus to demonstrate against (the sample paper's own diagrams). Video-RAG covers the temporal/video-specific case separately.

Hands-on

The full comparison below is runnable, against the sample paper's own real embedded figures — not stock images: Multimodal RAG → in the code repo. Both indexing approaches are fully local; only the final answer generation needs an LLM key.

TL;DR — quick revision
  • Two architectural approaches exist: a unified embedding space (a CLIP-family model puts text and images in the same vector space, so one similarity search covers both) or separate per-modality indexes combined by fusion
  • Newer isn't automatically better — measure it: 3 candidate embedding models tested on the same corpus and questions. SigLIP (often cited as CLIP's successor) scored worse than base CLIP on both text and image retrieval. jina-clip-v2 doubled base CLIP's text Recall@3 (0.375 → 0.75) at the same image accuracy, and is the model used throughout this page
  • The unified approach has a real cost, even with the best model tested: routing text through jina-clip-v2's text encoder instead of a dedicated text embedding model costs 25 points of Recall@3 on 8 factual questions, shown in full below
  • Separate indexes + RRF fusion is the better default, verified two ways: it correctly retrieves both a diagram and its explanatory text for an image-appropriate question, and correctly returns only text for a plain factual question
  • RRF is rank-based, not relevance-based: with a small image pool, the top-ranked image can win fusion on pure noise unless gated by a minimum-similarity threshold; and RRF produces an exact tie at rank 0 between modalities on every query — a structural property of rank fusion that no tiebreaker resolves

Two ways to combine modalities

Approach 1 — unified embedding space. A model like CLIP embeds text and images into the same shared vector space, so a text query and a relevant image end up close together despite being different modalities entirely. One similarity search covers everything.

Approach 2 — modality-specific indexes + fusion. Keep separate indexes per modality — text through a dedicated text embedder, images through CLIP — retrieve from each independently, then combine the results. The same idea as hybrid sparse+dense retrieval — combining two differently-strengthed retrieval signals rather than trusting one to cover everything — just across modalities here instead of retrieval methods.

The obvious question is which one to actually reach for. Testing both against the same real corpus gives a clear, measured answer.

Is there something better than CLIP?

CLIP (2021) is genuinely old by ML standards. Rather than guess at a replacement, the MTEB multimodal retrieval leaderboard is the real place to check — its top entries (Qwen3-VL-Embedding-8B, Gemini Embedding 2, Voyage Multimodal 3.5) are either too large for an 8GB machine to run locally or API-only, so 2 realistically-sized, locally-runnable candidates got measured head-to-head against base CLIP, on the exact same corpus and questions:

  • SigLIP (Google, often cited as the strongest current open-weights image-text model) — loaded fine, but scored worse on both axes: 0.25 Recall@3 on plain text (vs. CLIP's 0.375) and correctly matched only 1 of 4 test queries to its diagram (vs. CLIP's 2 of 4), with near-zero, uncalibrated raw similarity scores throughout. SigLIP is trained with a different loss (sigmoid, not contrastive softmax) and a much shorter text context window — likely a genuine mismatch for this use case, not a config error.
  • Jina CLIP v2 (built specifically for long-context text+image retrieval, including charts/tables) — needs transformers==4.46.3 pinned, since newer transformers releases dropped a function its custom model code depends on, and trust_remote_code=True, since it ships custom modeling code from its own HuggingFace repo. Its sentence-transformers integration doesn't route image inputs correctly, so this recipe calls the model's own AutoModel.encode_text / encode_image methods directly instead of the generic .encode() wrapper used everywhere else on this site.

Recall@3 and image-matching accuracy for all three, on the same corpus and questions:

Model                                Text Recall@3   Image queries correct (of 4)
---------------------------------------------------------------------------------
clip-ViT-B-32 (base CLIP)            0.375           2/4
google/siglip-base-patch16-224       0.25            1/4
jinaai/jina-clip-v2                  0.75            2/4   <- winner

jina-clip-v2 is what this page's code and every example below use — it doubles base CLIP's text-retrieval quality at the same image-matching accuracy. It isn't flawless: it still gets 2 of the 4 image-specific test queries wrong — asked about scaled dot-product attention specifically, it retrieves the Multi-Head Attention diagram instead (the two diagrams are visually and conceptually close). The broader lesson: a model being newer or more cited as SOTA (SigLIP) doesn't automatically mean it's better for a given use case — only measuring each candidate settles it.

More memory available? Try a bigger model

jina-clip-v2 was chosen to fit an 8GB machine. With more RAM/VRAM, Qwen3-VL-Embedding-8B (top of the MTEB multimodal leaderboard at the time of writing, but needing well over 8GB) is worth measuring the same way. Gemini Embedding 2 and Voyage Multimodal 3.5 are API-only alternatives — no local memory cost, but they need MultimodalEncoder rewritten to call the provider's endpoint instead of a local transformers model.

Building both, against the paper's own real figures

The sample "Attention Is All You Need" PDF has 3 embedded diagrams — the overall encoder-decoder architecture, the scaled dot-product attention mechanism, and multi-head attention — extracted directly, not substituted with stock images:

from pathlib import Path

from PIL import Image


def extract_images_from_pdf(pdf_path: str) -> dict[str, Image.Image]:
    """Pulls the paper's own real embedded figures out of the PDF —
    not stock images, the actual diagrams the authors put in the
    document."""
    from pypdf import PdfReader

    reader = PdfReader(pdf_path)
    images = {}
    for page_idx, page in enumerate(reader.pages):
        for img in page.images:
            key = f"page{page_idx}_{Path(img.name).stem}"
            images[key] = img.image
    return images

The unified approach

Everything — text chunks and images alike — goes into one shared embedding space (jina-clip-v2, the winner measured above), so a single similarity ranking can return either kind of result:

import numpy as np


def build_unified_index(chunks: list[str], images: dict, clip_model) -> dict:
    """Everything — text chunks AND images — embedded into ONE shared
    CLIP-family space. Enables genuine cross-modal search (a text
    query can directly retrieve an image), at a real cost: even
    jina-clip-v2 (see multimodal_encoder above), the strongest of 3
    text+image models tested, is still a weaker text retriever than a
    dedicated text embedding model (measured: 1.00 -> 0.75 Recall@3 on
    the same 8 factual questions, same corpus — see multimodal_rag.py's
    --compare-text-quality)."""
    text_embs = clip_model.encode(chunks, convert_to_numpy=True, show_progress_bar=False)
    image_names = list(images.keys())
    image_embs = clip_model.encode(list(images.values()), convert_to_numpy=True, show_progress_bar=False)
    return {"chunks": chunks, "chunk_embs": text_embs, "image_names": image_names, "image_embs": image_embs}


def unified_search(query: str, index: dict, clip_model, k: int = 3) -> list[tuple[str, str]]:
    query_vec = clip_model.encode(query, convert_to_numpy=True)
    candidates = [("text", c) for c in index["chunks"]] + [("image", n) for n in index["image_names"]]
    all_embs = np.vstack([index["chunk_embs"], index["image_embs"]])
    sims = [float(np.dot(query_vec, v) / (np.linalg.norm(query_vec) * np.linalg.norm(v) + 1e-8)) for v in all_embs]
    top_indices = np.argsort(sims)[::-1][:k]
    return [candidates[i] for i in top_indices]

Measured: a real cost to plain-text retrieval, even with the best model tested

Using a CLIP-family text encoder (required if text and images share one space) means giving up the dedicated text embedding model used everywhere else on this site. Measured directly, on 8 real factual questions against this same corpus, using jina-clip-v2 specifically — not the weaker base CLIP:

Question                                          Keyword         Naive   jina-clip-v2
------------------------------------------------------------------------------------
How many attention heads did they use?            "h = 8"         ✅      ✅
What is the model's embedding dimension?          "dmodel = 512"  ✅      ❌
How many layers are in the encoder?               "N = 6"         ✅      ✅
What optimizer was used for training?             "Adam"          ✅      ✅
What BLEU score did they get (En-De translation)? "28.4"          ✅      ✅
What GPUs was the model trained on?                "P100"         ✅      ✅
How long did the base model train for?            "12 hours"      ✅      ✅
What dropout rate did they use?                   "Pdrop = 0.1"   ✅      ❌

Recall@3:                                                         1.000   0.750

That's 25 points of Recall@3 lost, on plain factual questions with no image relevance at all — smaller than base CLIP's 62-point gap, but not zero, even with the model actually built for long-context text+image retrieval. The cost isn't a training-data mismatch specific to one model; it's structural: any text encoder good enough to sit in the same space as image embeddings is trading away some text-specific retrieval quality to do it. That's the real reason to reach for separate indexes whenever text quality matters and images don't need to directly outrank text.

Separate indexes + RRF fusion

Text goes through the dedicated embedder (full retrieval quality preserved); images go through CLIP (the only place CLIP is actually needed); the two independently-ranked lists get combined with Reciprocal Rank Fusion — the same RRF used for multi-query/RAG-Fusion elsewhere on this site:

import numpy as np
from sentence_transformers import SentenceTransformer

# RRF has no concept of absolute relevance — it only knows RANK within
# a list. With a small image pool, whichever image scores even
# marginally highest still becomes "rank 1" and earns a real fusion
# boost, even when every image is genuinely irrelevant to the query
# (verified directly: all 3 images scored well below this threshold —
# noise, not signal — for a plain-text question, and without a floor
# the highest of the three would still get fused ahead of a relevant
# text chunk). A minimum similarity floor fixes it: don't let a
# modality contribute to fusion at all if its best candidate doesn't
# clear a real relevance bar.
#
# A second, deeper property this surfaced: even with the floor in
# place, RRF still produces an exact tie at rank 0 between modalities
# on every query (1/(k+0+1) is identical for a text chunk or an image
# in that slot) — a structural fact about rank-based fusion, not
# something a tiebreaker can fix, since the top item in any list
# normalizes to the same value by construction. RRF is a consensus
# mechanism, not a magnitude comparator; see unified_search elsewhere
# in this file for how to get a genuine image-only result instead.
CLIP_RELEVANCE_THRESHOLD = 0.28


def build_separate_indexes(chunks: list[str], images: dict, text_model: SentenceTransformer, clip_model) -> dict:
    """Text goes through a DEDICATED text embedder (full retrieval
    quality preserved); images go through the CLIP-family model (the
    only place it's actually needed). Two separate, uncalibrated
    embedding spaces — which is exactly why fusion below uses RRF
    (rank-based) rather than comparing raw similarity scores
    directly."""
    return {
        "chunks": chunks,
        "chunk_embs": text_model.encode(chunks, convert_to_numpy=True, show_progress_bar=False),
        "image_names": list(images.keys()),
        "image_embs": clip_model.encode(list(images.values()), convert_to_numpy=True, show_progress_bar=False),
    }


def reciprocal_rank_fusion(ranked_lists: list[list], k: int = 60) -> list:
    """Same RRF used for multi-query/RAG-Fusion elsewhere in this
    repo — rank-based, not raw-score-based, which matters here since
    the CLIP-family model's image-similarity scores and the text
    embedder's similarity scores come from two different,
    uncalibrated spaces and can't be compared directly.

    A real, non-obvious consequence: EVERY query produces an exact
    tie at rank 0 between modalities — `1/(k+0+1)` is identical
    whether a text chunk or an image sits in that slot, regardless of
    how relevant either actually is. This isn't a bug to patch with a
    tiebreaker (a first attempt at one — normalizing each candidate's
    raw similarity within its own list — didn't work: the top item in
    ANY list normalizes to exactly 1.0 by construction, so the tie
    persists no matter what). It's a real, structural property of
    rank-based fusion: RRF is a CONSENSUS mechanism, not a magnitude
    comparator, and it has no principled way to let one modality's
    confident top pick outright beat another modality's merely-present
    top pick. See `unified_search` above for how to get a genuine
    image-only result instead — a single flat ranking has no second
    list to tie against."""
    scores: dict = {}
    for ranked in ranked_lists:
        for rank, item in enumerate(ranked):
            scores[item] = scores.get(item, 0.0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=lambda item: -scores[item])


def fused_search(query: str, index: dict, text_model: SentenceTransformer, clip_model, k: int = 3) -> list[tuple[str, str]]:
    text_query_vec = text_model.encode(query, convert_to_numpy=True)
    text_sims = [float(np.dot(text_query_vec, v) / (np.linalg.norm(text_query_vec) * np.linalg.norm(v) + 1e-8)) for v in index["chunk_embs"]]
    text_ranked = [("text", index["chunks"][i]) for i in np.argsort(text_sims)[::-1]]

    clip_query_vec = clip_model.encode(query, convert_to_numpy=True)
    image_sims = [float(np.dot(clip_query_vec, v) / (np.linalg.norm(clip_query_vec) * np.linalg.norm(v) + 1e-8)) for v in index["image_embs"]]
    ranked_lists = [text_ranked]
    if max(image_sims, default=0.0) >= CLIP_RELEVANCE_THRESHOLD:
        image_order = np.argsort(image_sims)[::-1]
        ranked_lists.append([("image", index["image_names"][i]) for i in image_order if image_sims[i] >= CLIP_RELEVANCE_THRESHOLD])

    fused = reciprocal_rank_fusion(ranked_lists)
    return fused[:k]

Measured: correctly retrieves the right modality for each question type

Image + text, mixed correctly — asked "How are multiple attention heads concatenated together?" (deliberately not the scaled-dot-product query — jina-clip-v2 gets that one wrong, see above; this is the one verified correct):

[TEXT]  "...Figure 5: Many of the attention heads exhibit behaviour
         that seems related to the structure of the sentence..."
[IMAGE] Multi-Head Attention diagram (multiple parallel attention
         heads, Linear, Concat)
[TEXT]  "...output values. These are concatenated and once again
         projected, resulting in the final values, as depicted in
         Figure 2. Multi-head attention allows the model to jointly
         attend to information from diff[erent subspaces]..."

Answer: "...The outputs from each individual attention head (head_1,
         head_2, ..., head_h) are concatenated and then projected
         through a linear transformation using a weight matrix W^O.
         MultiHead(Q,K,V) = Concat(head_1,...,head_h)W^O..." —
         grounded in both the diagram and the surrounding text.

Text only, correctly, for a plain factual question — asked "What optimizer did they use for training?":

[TEXT]  "warmup_steps = 4000. 5.4 Regularization..."
[TEXT]  "(3.5 days). 5.3 Optimizer We used the Adam optimizer [20]
         with β1 = 0.9, β2 = 0.98 and ϵ = 10−9..."
[TEXT]  "target tokens. 5.2 Hardware and Schedule We trained our
         models on one machine with 8 NVIDIA P100 GPUs..."

Answer: "...they used the Adam optimizer for training, with
         β1 = 0.9, β2 = 0.98, ϵ = 10^-9." — no image pulled in,
         because none is actually relevant here.

RRF has no relevance floor

RRF only knows RANK within a list — it has no concept of whether "rank 1" is actually relevant or just the least-irrelevant option available. With only 3 images in the pool, all three can score a nearly-identical, low similarity against a plain text question — genuine noise, not signal — yet the highest of the three still becomes "rank 1," earns a real RRF fusion score, and gets pulled into the final top-3 ahead of a relevant text chunk. A minimum similarity threshold fixes this: a modality's results don't contribute to fusion at all unless its best candidate clears a real relevance bar. This is the same effect multi-query/RAG-Fusion demonstrates elsewhere on this site — RRF diluting a good match with weak competition — showing up here as a small-candidate-pool edge case instead.

Why there's no "image-only" example on this page

It's a fair question to ask for one. Against several deliberately image-specific queries ("a diagram showing MatMul Scale Mask SoftMax boxes", "Linear Concat multi-head attention diagram"), neither approach — with base CLIP or with jina-clip-v2 — returns an image-only top-3 result on this corpus. There are two structural reasons why:

Reason 1 — RRF's rank-0 tie (separate + fusion approach). 1/(k+0+1) is mathematically identical whether a text chunk or an image occupies rank 1 of its own list — a genuine tie, not a near-miss. Normalizing each candidate's raw similarity within its own list, then breaking ties by that, doesn't help either: the top item in any list normalizes to exactly 1.0 by construction, so the tie persists regardless. RRF is fundamentally a consensus mechanism across ranked lists, not a magnitude comparator, so it has no principled way to let one modality's confident pick outright beat another's merely-present one.

Reason 2 — the modality gap (unified approach), present but smaller with a better model. With base CLIP, all 3 images land at the very bottom of the combined 95-item ranking (positions 92-94) for every image-specific query — a stark, uniform floor. With jina-clip-v2, that gap narrows substantially for at least one image: for "a diagram showing MatMul Scale Mask SoftMax boxes", the Scaled Dot-Product Attention diagram ranks 4th of 95 overall. That's still not the same as landing in unified_search's literal top-3, though — in practice it lands at position 5, just behind four text chunks whose wording overlaps the query more tightly ("queries and keys of dimension dk...", "Attention(Q,K,V) = softmax(QK^T/√dk)V"). Even a well-positioned image doesn't out-rank text passages that are near-verbatim matches to the query's own phrasing.

This matches a real, published, well-studied phenomenon: CLIP-family text and image embeddings occupy structurally separate regions of the shared vector space — an artifact of how contrastive vision-language models are initialized and trained, not specific to this corpus or model. jina-clip-v2 narrows that gap compared to base CLIP but doesn't close it — cross-modal similarity (text query vs. image) is still systematically weaker than intra-modal similarity (text query vs. other text), even when the image is the genuinely correct answer.

Takeaway: a reliable image-only result on a corpus this size, with either approach here, would need a much larger, more distinctly-imaged corpus, a model with an even smaller modality gap than jina-clip-v2, or an explicit modality-routing step upstream (the same routing idea covered on the Modular RAG page) rather than relying on fusion or a shared embedding space alone.

When to use which approach

Use separate indexes + fusion for: the general case — it preserves full text-retrieval quality and only pays the CLIP-family model's cost where it's actually needed (images), and correctly returns text-only results for plain factual questions. This should be your default. Use a unified embedding space for: specifically when you need images to be able to outrank text outright in a genuinely open, single ranking — RRF's rank-0 tie structurally prevents that in the fused approach, so if your use case depends on an image sometimes winning cleanly over mediocre text matches, unified is the only one of the two that can do it (at the real text-quality cost measured above, 25 points of Recall@3 even with the best model tested). Skip multimodal retrieval entirely for: a corpus that's genuinely text-only — the added complexity (a second embedding model, a fusion or unification step) buys nothing if there's no second modality to retrieve.

Scenario Check

Sources & further reading