Sourcegraph Cody: Abandoning Embeddings for Search, and Admitting What They Can't Measure¶
Sourcegraph's Cody is a coding assistant that has to retrieve relevant context from codebases ranging in size from roughly a hundred repositories to over a hundred thousand, across features — chat, autocomplete, edit, test generation — that each run under a completely different latency budget. What makes Cody's public writeups worth studying isn't just the "Context Engine" architecture itself, but a real reversal at its center: Cody launched with embeddings as its core retrieval mechanism and then explicitly walked that decision back, for reasons that had nothing to do with embeddings being inaccurate.
The problem¶
A codebase isn't one thing to retrieve over. A hundred-repo customer and a hundred-thousand-repo customer have fundamentally different scaling requirements. Autocomplete has to return context in the time it takes to not interrupt typing; chat can afford to spend more time assembling a richer answer. Building one retrieval pipeline that served all of this well, at enterprise scale, while keeping customers' proprietary code where they wanted it, turned out to require different strategies for different features rather than one universal pipeline.
What they built, and why¶
Cody's retrieval layer is a two-stage Context Engine. The first stage retrieves candidates from multiple heterogeneous sources, optimized purely for recall — cast a wide net. The second stage runs a pointwise ranker model that scores each candidate for relevance, and instead of taking a fixed top-k, it applies a token-budget and threshold cutoff to decide what actually makes it into the final prompt — the number of chunks included is a function of relevance quality and available context budget, not an arbitrary fixed count.
On top of that shared engine, each feature uses its own retrieval strategy rather than one pipeline serving all of them. Chat blends keyword search — BM25 plus tuned relevance signals over Sourcegraph's own code-search index — with semantic embeddings, explicit @-mention control so a developer can point directly at a file or symbol, and external connectors through OpenCtx for sources like Slack, Notion, and Linear. Autocomplete takes a deliberately different path and avoids embeddings entirely, relying instead on Jaccard similarity over recently opened files, Tree-sitter-based structural parsing of the code itself, and Reciprocal Rank Fusion to combine these signals — because embeddings add latency that autocomplete's interruption-free budget can't absorb.
The central story, though, is what happened to embeddings as the primary retrieval path. Cody originally launched on OpenAI's text-embedding-ada-002 (1536 dimensions) as its core retrieval mechanism, and Sourcegraph later walked that back for three explicit reasons. First, sending proprietary customer code to a third party for embedding was a real security and privacy blocker for enterprise customers — not a hypothetical concern, but the actual objection that came up in enterprise sales and deployment conversations. Second, the operational overhead of maintaining vector databases and keeping them continuously fresh as codebases changed was a genuine ongoing cost, not a one-time setup expense. Third, and most fundamentally, vector search doesn't scale cleanly past roughly a hundred thousand repositories — at Sourcegraph's largest customer scale, the approach that looked best in a smaller proof-of-concept stopped being viable. Sourcegraph replaced embeddings as the primary path with native BM25-based Sourcegraph Search plus code-graph signals, keeping embeddings around only as an optional, local mode rather than the backbone of the system.
Sourcegraph's own arXiv paper is also unusually candid about evaluation, which is worth treating as a finding in its own right. The paper states plainly that they report no hard quantitative retrieval metrics from production, for three structural reasons: there's a large online/offline gap, because fully logging a user's live workspace state for offline evaluation would carry major privacy and engineering costs; there's a severe shortage of labeled data, because labeling retrieval quality well requires deep, codebase-specific expertise that generic crowdsourcing can't provide; and the system has multiple interacting components — retrieval, ranking, the LLM itself — which makes it genuinely hard to attribute an end-to-end quality change to any single stage. Their workaround is a set of domain-specific proxy checks — syntactic parse validity, type-checking, test execution, symbol-existence checks — used both as offline evaluation signals during development and as live guardrails in production.
The numbers¶
Cody operates across a repository scale of roughly 100 to over 100,000 repos, with up to 10 repos selectable per chat query. The now-deprecated ada-002 embedding path used 1536-dimensional vectors. No production accuracy or latency numbers have been publicly disclosed for the Context Engine — a gap Sourcegraph states directly in its own source material, not an omission introduced here.
The lesson
RAG architecture decisions are frequently driven by security, compliance, and operational scaling limits rather than pure accuracy — Cody dropped the retrieval method that looked best on paper because it failed at enterprise data-governance and hundred-thousand-repo scale, a real counter-example to assuming the most sophisticated technique automatically wins. Just as valuable is Sourcegraph's candor about evaluation itself: admitting you lack solid online metrics because of the online/offline gap and label scarcity is a legitimate, underreported problem in production RAG systems, and treating proxy checks as both an offline signal and a live guardrail is a workable way to ship without pretending you have metrics you don't.