Skip to content

Video RAG

Every technique on this site so far treats a document as either text or a static image. Video adds a dimension neither has: time. A single frame or a fixed-length clip can miss context that only makes sense across a longer span — a graph being drawn stroke by stroke, an explanation that starts in one moment and lands in the next — and a query needs to find not just the right content, but the right moment.

Scope: this page builds the decomposed version, not native video understanding — read this before the rest

"Video RAG" spans three genuinely different things, and it matters which one you mean:

  1. Decomposed retrieval, decomposed generation (what this page builds). Speech-to-text produces a transcript; frames are sampled as still images; search only ever runs against the transcript text; generation gets one static frame plus transcript text, because the LLM used here (Claude) has no video input at all — its API accepts images and PDF documents, never a video file. This turns a video into a text-plus-image problem. It's real, it's what most practical open-source implementations actually do, and it's genuinely useful — but it is not a model perceiving video as video.
  2. Native "omni" models that take actual video (frames + synchronized audio) as one input — Gemini's video file API, Qwen2.5-Omni, GPT-4o's audio+vision path. Handed an actual clip, these can answer something like "what does he do right as he says X," which a single sampled frame plus separately-transcribed text structurally cannot, because motion and audio-visual timing were never decomposed away in the first place.
  3. Video-native retrieval — embedding whole clips directly (InternVideo, ViCLIP) instead of frame-by-frame CLIP, so even finding the right moment doesn't go through a text/image proxy (the approach in the VideoRAG paper, arXiv 2501.05874).

This page is tier 1, on purpose: tiers 2 and 3 need either a model this recipe's generation step (Claude) cannot use, or video-embedding models in the 1B+ parameter range — both unrealistic on an 8GB machine without a dedicated GPU. Confirmed directly against Claude's own API docs before writing this: there is no video content-block type, only image and document. If you swap the generation step for a model with native video input, tier 1's whole "sample frames, pick the nearest one" mechanism becomes unnecessary — that's a materially different recipe, not a bigger version of this one.

Hands-on

The full comparison below is runnable, against a real public-domain video: Video RAG → in the code repo. Transcription and retrieval are fully local; only generation needs an API key.

TL;DR — quick revision
  • Two signals, one timestamp axis: a speech-to-text transcript (Whisper) and frames sampled at a fixed interval, both indexed by time, so a query can retrieve the relevant moment's transcript AND see what was on screen at that moment
  • Measured directly: transcript-only retrieval scores 3/5 on a real eval set built from this page's own sample video — both misses have a distinct, checked cause, not a vague "retrieval sometimes fails"
  • One failure mirrors ordinary chunk-boundary problems, just on the time axis: Whisper's own segmentation split a cause from its explanation across two adjacent segments — retrieval found the segment ending in "...begins to warm up because," but the actual reason was in the next one
  • The other is embedding confusion between near-duplicates: two adjacent segments about similar-sounding percentages (90% vs. 100% of two different UV bands) crowded each other out of the top-3
  • There's a real, verified case transcript-only retrieval cannot solve at all: a whiteboard axis label that's shown but never spoken — the transcript-only path correctly declines rather than guessing, and only passing the actual frame image gets the right answer
  • This recipe deliberately doesn't index frames by visual similarity — frame grounding here is timestamp-proximity to the retrieved transcript segment, not a separate CLIP-based visual search (that mechanism is already covered on the Multimodal RAG and Vision RAG pages)

The sample video, and why it's a fair test

NASA's "NASA Explains Cold Zone Above Tropics" (public domain, CC0) — a real, 5-minute-12-second video, verified directly before using it. NASA scientist Paul Newman explains atmospheric layers (troposphere, tropopause, stratosphere) and how the ozone layer absorbs UV radiation, while drawing a live graph on a whiteboard. The video actually mixes several visual styles — an animated title card, animated illustrations, hand-lettered word overlays, and genuine live-action whiteboard footage — and critically, the whiteboard's own axis labels ("ALTITUDE," "TEMPERATURE") are never spoken anywhere in the narration. That's what makes it a fair test: there's a real, verifiable fact that only exists visually, not just a hypothetical one.

Pipeline: transcript and frames, both keyed by timestamp

Frames and transcript segments live on two different timelines, and they're linked by nothing more than arithmetic on a shared unit (seconds) — not a shared index, and not a joint embedding space.

Frames sit on a fixed, even grid. ffmpeg samples one frame every 5 seconds, so frame i's timestamp is always i × 5 — frame 0 at 0s, frame 1 at 5s, frame 2 at 10s, and so on, no exceptions, for the whole video.

Transcript segments sit on an uneven grid Whisper decides on its own. Segments are cut wherever Whisper detects a natural speech boundary — a pause, a sentence end — so their start times almost never land on a multiple of 5. A real segment from this video: {"start": 94.88, "end": 100.44, "text": "This cold point is called the tropopause."}.

Only the transcript is searched. The query is embedded once and compared by cosine similarity against every transcript segment's embedding — frames are never embedded against the query and never take part in the similarity search at all. The top-k result is just transcript segments, each carrying whatever start/end Whisper gave it.

Frames are matched to a search result afterward, by nearest timestamp — nothing more. Once a segment wins the search (say, the tropopause segment starting at 94.88s), the retrieved frame is whichever of the fixed 5-second samples is numerically closest to that start time:

segment start = 94.88s
frame timestamps = 0, 5, 10, ..., 90, 95, 100, ...
|94.88 - 90| = 4.88
|94.88 - 95| = 0.12   <- smallest, so the frame at 95s wins

That's the entire mechanism — min(frames, key=lambda pair: abs(pair[0] - timestamp)). The real consequence of it being this simple: since a frame exists only every 5 seconds, the one you get back is never guaranteed to be the exact right visual instant, just whichever sample happened to land closest. This worked for the whiteboard question below because the whiteboard stayed on screen for a while — any frame within a couple of seconds of it still showed the graph. A visual detail that only existed for a 1-2 second window between two samples could be missed entirely.

Frame extraction samples one frame every 5 seconds via ffmpeg — fixed-interval, not scene-detection, which is simple and adequate for a short video:

def extract_frames(video_path: str, interval_sec: int = 5) -> list:
    """Sample frames at a fixed interval via ffmpeg -- the only
    "extraction" step needed on the visual side. No scene detection, no
    frame-importance scoring; every Nth second gets a frame regardless of
    whether anything changed."""
    import subprocess
    import tempfile
    from pathlib import Path

    from PIL import Image

    with tempfile.TemporaryDirectory() as tmp:
        subprocess.run(
            ["ffmpeg", "-i", video_path, "-vf", f"fps=1/{interval_sec}", "-y", f"{tmp}/frame_%04d.jpg"],
            check=True, capture_output=True,
        )
        frames = []
        for i, path in enumerate(sorted(Path(tmp).glob("frame_*.jpg"))):
            frames.append((i * interval_sec, Image.open(path).copy()))
    return frames

Transcription produces timestamped segments. Whisper shells out to the system ffmpeg binary internally to decode audio, so it's a real prerequisite even though this function never calls it directly:

def transcribe_video(video_path: str, model_size: str = "small") -> list:
    """Speech-to-text with timestamped segments. Whisper shells out to
    the system `ffmpeg` binary internally to decode audio, so ffmpeg is a
    real prerequisite here even though this function never calls it
    directly."""
    import whisper

    model = whisper.load_model(model_size)
    result = model.transcribe(video_path, verbose=False)
    return [{"start": s["start"], "end": s["end"], "text": s["text"].strip()} for s in result["segments"]]

Retrieval over transcript segments uses the same embed-and-cosine-similarity pattern as plain text RAG elsewhere on this site — the only difference is each retrievable unit carries a timestamp instead of a page or chunk id:

def transcript_search(query: str, segments: list, k: int = 3) -> list:
    """Embed each transcript segment once, then rank by cosine
    similarity to the query -- the same pattern used for text chunks
    elsewhere on this site, just with a timestamp attached to each
    unit instead of a page or document id."""
    from sentence_transformers import SentenceTransformer, util

    model = SentenceTransformer("all-MiniLM-L6-v2")
    texts = [s["text"] for s in segments]
    segment_embeddings = model.encode(texts, convert_to_numpy=True, show_progress_bar=False)
    query_embedding = model.encode(query, convert_to_numpy=True)
    sims = util.cos_sim(query_embedding, segment_embeddings)[0]
    top_k = sims.topk(min(k, len(segments)))
    return [segments[i] for i in top_k.indices.tolist()]

Frame grounding is driven entirely by proximity to the retrieved segment's timestamp — frames only exist at the fixed 5-second samples, so this is always "the closest sampled frame," not a guarantee of the exact right visual instant:

def nearest_frame(timestamp: float, frames: list):
    """Frames only exist at the fixed sampling interval, so the frame
    actually shown alongside a retrieved transcript segment is whichever
    sampled frame is CLOSEST to that segment's start time -- not
    necessarily the exact right visual moment."""
    return min(frames, key=lambda pair: abs(pair[0] - timestamp))[1]

Measured: transcript-only Recall@3 — 3 of 5

A 5-question eval set built from this video's own real transcript:

PASS  What temperature range is found at ground level in the tropics?
PASS  What is the point called where the troposphere meets the stratosphere?
PASS  What percentage of UV-B radiation is absorbed by the ozone layer?
FAIL  What percentage of UV-C radiation is absorbed by the ozone layer?
FAIL  Why does the stratosphere begin to warm up?

Two misses, two distinct, checked causes

UV-C question. The exact right segment exists in the transcript — "There's another class called UVC... 100% of the UVC" — but it wasn't in the top-3. Two adjacent segments about UV-B's 90% figure and generic "absorbed by the ozone layer" phrasing crowded it out: three segments about percentages and ozone absorption look extremely similar to an embedding model, and the one with the actually-different number lost.

Stratosphere-warming question. Retrieval correctly found the segment ending in "...the reason the stratosphere begins to warm up is because" — and stopped there, because that's where Whisper's own segmentation drew the line. The actual reason ("...radiation from the Sun is being absorbed in the ozone layer") landed in the very next segment, which wasn't retrieved. This is the same shape as a text-chunking boundary problem — a cause and its explanation split across two retrieval units — just happening on the time axis instead of the character-count axis.

The case transcript-only retrieval cannot answer

Asked "What does the Y-axis of the whiteboard graph say?" — a detail that's shown on screen but genuinely never spoken:

Transcript-only answer: The transcript doesn't specify what the Y-axis of
the whiteboard graph says.

With-frame answer: Based on the image, the Y-axis of the whiteboard graph
appears to say "ALTITUDE" (written vertically along the left side of the
graph).

The transcript-only path doesn't hallucinate a plausible-sounding label — it correctly says the answer isn't in the given context. That's the right behavior with only text to work from. Passing the actual frame, retrieved by proximity to the closest relevant transcript segment, is what makes the question answerable at all. This is the direct, video-specific version of the same principle demonstrated on the Multimodal RAG page: some answers are visual facts, not textual ones, and no amount of better text retrieval recovers information that was never in the text to begin with.

What this recipe leaves out, on purpose

There's no separate visual (CLIP-based) frame search here — frame grounding is timestamp proximity to whichever transcript segment retrieval already found, not a query embedded against frame images directly. A fuller pipeline could add that: embed each frame with a CLIP-family model (exactly the mechanism already built and measured on the Multimodal RAG and Vision RAG pages) to catch moments where the relevant visual doesn't line up with any well-matching transcript segment at all — a purely visual event with no narration over it, for instance. Left out here to keep this page focused on the timestamp-linking mechanism that's specific to video, rather than re-deriving image-embedding choices already covered elsewhere on this site.

When to use which

Transcript-only retrieval is enough when: the answer is something that gets said out loud, and the video's narration reliably covers its own content — most explainer/lecture-style videos. Add frame grounding when: the video shows things it doesn't narrate — on-screen labels, diagrams, text, or visual detail that matters to the likely questions. Consider a full CLIP-based visual search (not built here) when: a meaningful fraction of the video's important content has no narration over it at all, so timestamp-proximity to a transcript hit isn't a reliable way to find the right frame.

Scenario Check

Sources & further reading