Skip to content

Vector Databases

Everything upstream of this page — chunking, embedding, hybrid retrieval — assumes there's somewhere to actually store and search the vectors. This page covers that storage layer in full: the ANN algorithms and every parameter they expose, real tested code against six real databases sharing one corpus and one metadata scheme, and the system-design reasoning interviewers actually probe when they ask "how would you choose a vector database for this."

Hands-on

The full pipeline below is runnable: Vector Databases → in the code repo. No API key needed — embeddings run locally.

TL;DR — quick revision
  • The core problem is approximate, not exact, nearest-neighbor search, and there are exactly three algorithm families worth knowing in depth: HNSW (graph-based, the default), IVF (cluster-based, needs training), and PQ (compresses the vectors themselves, usually paired with IVF)
  • Every tunable parameter trades off recall against speed, memory, or build time — M/ef_construction/ef_search for HNSW, nlist/nprobe for IVF, m/nbits for PQ — this page explains what each one literally does, with real current defaults from each database's own source or docs
  • Six real databases, same corpus, same metadata, both HNSW and IVF shown as runnable code — FAISS, ChromaDB, Qdrant, Milvus Lite, pgvector, and Weaviate, all indexed and queried against the same chunked PDF with each chunk's real page number as metadata
  • Choosing a vector database is a system design question, not a feature-comparison exercise — the standard senior-level answer starts with "do you need one at all," reasons through a recall/latency/memory triangle (plus quantization as a fourth lever), and only escalates to a dedicated store once a concrete constraint demands it

Why vector databases exist

The core problem: efficient similarity search over millions to billions of high-dimensional vectors. Brute-force comparison — the query vector against every stored vector — is O(n) per query, impractical at any real scale. Vector databases solve this with Approximate Nearest Neighbor (ANN) search: trading a small amount of accuracy (occasionally missing the true top-1) for a large speed gain, since "very close to the true top-k" is what RAG retrieval actually needs.

There are three ANN algorithm families worth knowing in real depth, because nearly every vector database's index configuration is just a different naming of the same underlying knobs.

HNSW: the graph-based default

Mechanism. HNSW (Hierarchical Navigable Small World) builds a multi-layer graph. Every vector lives in layer 0; each vector is also inserted into some number of layers above it with exponentially decaying probability, so the top layer has very few nodes with long-range connections and lower layers get progressively denser. A search starts at a single entry point in the top layer, greedily moves toward the query's region, descends layer by layer — a "zoom-in" phase similar in spirit to a skip list — and only does the expensive, wide candidate search once it reaches layer 0. Each node's neighbor list is capped: M connections per layer above 0, and a separate, larger cap (2M) at layer 0 specifically, since every vector lives there and needs more connectivity to stay well-linked.

Parameters, what they actually control, and how to choose them (per the original paper — Malkov & Yashunin, IEEE TPAMI 2018, arXiv:1603.09320 — and current vendor documentation):

Parameter What it controls Trade-off Tunable after build?
M (max_neighbors, max_connections, maxConnections depending on library) Max bidirectional edges per node per layer Higher M → better recall, especially at high dimensionality or high target recall, at the cost of more memory (roughly linear in M) and slower builds. The paper's own guidance: "a reasonable range of M is from 5 to 48... smaller M generally produces better results for lower recalls and/or lower dimensional data, while bigger M is better for high recall and/or high dimensional data" No — baked into the graph structure at build time
ef_construction Size of the dynamic candidate list used while choosing which neighbors to actually link, during insertion Higher → better graph quality → better recall ceiling for every future search, at the cost of slower index builds. The paper found diminishing returns fast: "a reasonable quality index can be constructed for efConstruction = 100... in just 3 minutes. Further increase... leads to little extra performance but in exchange for significantly longer construction time" No — also baked in at build time
ef_search (ef, efSearch, hnsw_ef) Size of the dynamic candidate list at query time Higher → more of the graph explored → better recall, higher latency. Must be ≥ the number of results requested. This is the knob to sweep first when tuning recall, since it needs no rebuild Yes — a pure query-time setting

Real current defaults, pulled directly from each library's source or docs (not blog posts, which are often stale):

Library M default ef_construction default ef_search default
FAISS (IndexHNSWFlat) 32 40 16
ChromaDB (max_neighbors) 16 100 100
Qdrant (m) 16 100 = ef_construct unless overridden per-query
pgvector 16 64 40
Weaviate (max_connections) 32 128 -1 (dynamic — see below)
Milvus no fixed default (valid range 2–2048; leans on AUTOINDEX) no fixed default (range 1–INT_MAX) must be ≥ top_k

Every demo on this page sets M=32, ef_construction=200, ef_search=64 uniformly across all six databases (above most of their defaults, since this corpus is small enough that build cost is irrelevant and the point is showing higher-quality search).

Weaviate's dynamic ef mode

Weaviate's ef defaults to -1, meaning "dynamic" — it computes an effective ef per query as dynamic_ef_factor × limit, clamped between dynamic_ef_min (100) and dynamic_ef_max (500), rather than using one fixed value for every query regardless of how many results were actually requested.

Mechanism. Run k-means over a training sample of the dataset to produce nlist centroids, partitioning the vector space into Voronoi cells. Every vector is then assigned to (and stored in) the inverted list of its nearest centroid — hence "inverted file index." At query time, only the nprobe cells closest to the query are actually scanned, instead of the whole dataset.

nlist — the number of clusters. FAISS's own sizing guidance (faiss wiki: Guidelines to choose an index): roughly 4·sqrt(N) to 16·sqrt(N) for N up to about a million vectors; past that, FAISS's own recipes switch to fixed power-of-2 sizes paired with an HNSW coarse quantizer for fast centroid lookup (IVF65536_HNSW32 for 1M–10M vectors, IVF262144_HNSW32 for 10M–100M, IVF1048576_HNSW32 for 100M–1B). Training needs a real, sufficiently large sample: FAISS's own guidance is 30× to 256× nlist training vectors — too few and the centroids come out unbalanced, starving some clusters and overloading others.

nprobe — how many of the nlist clusters get searched per query, the direct IVF equivalent of ef_search: default 1, tunable per-query with no rebuild, and search time scales roughly linearly with it. Setting nprobe = nlist is mathematically equivalent to brute-force search over the whole dataset — correct, but with none of IVF's speed benefit left.

pgvector supports IVFFlat directly (lists = nlist, probes = nprobe), with its own sizing rule in the README: rows / 1000 for under a million rows, sqrt(rows) above that.

FAISS IVF and pgvector IVFFlat, as code

def demo_faiss_ivf(chunks: list, embeddings, query: str, embed_query, nlist: int = 8) -> None:
    """IVF clusters vectors around `nlist` centroids (learned via
    k-means in `.train()`) and only searches the `nprobe` closest
    clusters at query time -- `nprobe=nlist` is equivalent to
    exhaustive search over every cluster. IVF's benefits only show up
    at real scale (thousands to millions of vectors); at a small corpus
    the clusters are too sparse to demonstrate a real speed/recall
    trade-off, though the code and training step are identical to
    production usage."""
    import faiss
    import numpy as np

    dim = embeddings.shape[1]
    page_by_id = {c["id"]: c["page"] for c in chunks}
    ids = np.array([c["id"] for c in chunks], dtype="int64")

    quantizer = faiss.IndexFlatIP(dim)
    ivf = faiss.IndexIVFFlat(quantizer, dim, nlist, faiss.METRIC_INNER_PRODUCT)
    ivf.train(embeddings)  # k-means over the vectors themselves; required before add()
    index = faiss.IndexIDMap2(ivf)
    index.add_with_ids(embeddings, ids)

    query_vec = embed_query(query).reshape(1, -1).astype("float32")

    ivf.nprobe = 1
    scores, result_ids = index.search(query_vec, 3)
    print("nprobe=1:", [(page_by_id[cid], round(float(s), 3)) for s, cid in zip(scores[0], result_ids[0]) if cid != -1])

    ivf.nprobe = nlist  # exhaustive -- searches every cluster
    scores, result_ids = index.search(query_vec, 3)
    print(f"nprobe={nlist} (exhaustive):", [(page_by_id[cid], round(float(s), 3)) for s, cid in zip(scores[0], result_ids[0]) if cid != -1])

def demo_pgvector_ivfflat(chunks: list, embeddings, query: str, embed_query, dsn: str) -> None:
    """pgvector supports IVFFlat as an alternative to HNSW. `lists` is
    sized per the pgvector README's own guidance: rows/1000 for under
    1M rows, sqrt(rows) above that."""
    import psycopg
    from pgvector.psycopg import register_vector

    dim = embeddings.shape[1]
    lists = max(1, len(chunks) // 1000)
    with psycopg.connect(dsn, autocommit=True) as conn:
        register_vector(conn)
        conn.execute("DROP TABLE IF EXISTS attention_paper_ivf")
        conn.execute(f"CREATE TABLE attention_paper_ivf (id BIGINT PRIMARY KEY, page INT, embedding VECTOR({dim}))")

        with conn.cursor() as cur:
            cur.executemany(
                "INSERT INTO attention_paper_ivf (id, page, embedding) VALUES (%s, %s, %s)",
                [(c["id"], c["page"], embeddings[i]) for i, c in enumerate(chunks)],
            )

        conn.execute(f"CREATE INDEX ON attention_paper_ivf USING ivfflat (embedding vector_cosine_ops) WITH (lists = {lists})")
        conn.execute("SET ivfflat.probes = 1")

        query_vec = embed_query(query)
        rows = conn.execute(
            "SELECT page, embedding <=> %s AS distance FROM attention_paper_ivf ORDER BY embedding <=> %s LIMIT 3",
            (query_vec, query_vec),
        ).fetchall()
        print(f"lists={lists}, probes=1:", [(page, round(d, 3)) for page, d in rows])

IVF needs real scale to show its actual behavior

Run against this page's 121-chunk corpus, FAISS's own training step prints WARNING clustering 121 points to 8 centroids: please provide at least 312 training points — a direct, real instance of the 30×–256× training-data rule above (8 clusters calling for roughly 240–2,048 training points, far more than this corpus has). At this scale nprobe=1 and nprobe=8 (exhaustive) return identical results, and pgvector's lists formula rounds down to 1 — a single cluster containing every row, which makes IVFFlat behave identically to a full scan. None of this is a bug in the code; it's IVF's clustering having nothing meaningful to do below the thousands-of-vectors range it's actually built for.

PQ: compressing the vectors themselves

Mechanism. Product Quantization splits each vector into m equal-length sub-vectors, then runs k-means independently within each of the m subspaces to learn a codebook of 2^nbits centroids. Each sub-vector is stored as just the ID of its nearest sub-centroid — the full vector compresses from D floats down to m small integer codes. Distances are approximated at query time via precomputed lookup tables between the query's sub-vectors and each subspace's centroids, without ever reconstructing the full-precision vector.

  • m (number of sub-quantizers): the vector dimension must be a multiple of m. More sub-quantizers generally means better reconstruction accuracy at a fixed nbits, at the cost of more codes to store per vector.
  • nbits (bits per sub-quantizer code): controls centroid count per subspace (2^nbits). 8 is the standard default specifically because 256 centroids fit in exactly one byte per sub-vector — clean byte alignment that's both memory-efficient and fast for SIMD-based lookup.

Concrete compression math (verified against Pinecone's product quantization writeup): a 128-dimensional float32 vector is 512 bytes uncompressed. Under PQ with m=8, nbits=8, it compresses to 8 sub-codes × 8 bits = 8 bytes — a 64× compression ratio. On the standard SIFT1M benchmark (1M × 128-dim vectors), this takes a FlatL2 index from 256MB down to 6.5MB. PQ is usually combined with IVF as "IVF-PQ" — IVF narrows the search to a few clusters, PQ makes each stored vector cheap enough that billion-scale corpora fit in RAM at all.

None of this page's own demos use PQ — at 121 vectors, compression has no benefit and would only add approximation error. It matters once memory, not latency, becomes the binding constraint (see the scale bands below).

Beyond RAM: disk-based ANN

Both HNSW and IVF-PQ assume the index lives in memory. Past a certain scale, it doesn't fit — Uber's own engineering writeup on their 1.5-billion-item HNSW deployment states the constraint directly: the graph has to be fully RAM-resident, or query latency degrades to "tens of seconds." Two disk-based designs exist specifically for this: DiskANN (Microsoft Research) keeps a compressed, PQ-style summary of every vector in RAM for fast candidate filtering, and fetches full-precision vectors from SSD only as needed — its own published SIFT-1B benchmark reports roughly 5,000 QPS at 95%+ recall on a single 64GB-RAM machine. SPANN (also Microsoft) takes the inverted-file idea further: centroids stay in RAM, the (much larger) posting lists live on disk — reported running in production at Bing, at hundreds-of-billions scale, using roughly a tenth of the RAM a comparable fully in-memory system would need. Milvus exposes DiskANN as a selectable index type directly; ChromaDB's distributed/cloud tier uses SPANN internally.

Similarity metrics

  • Cosine similarity: angle between two vectors, ignoring magnitude — the common default for text embeddings, since magnitude rarely carries meaningful signal there.
  • Dot product: factors in magnitude — used when a model was specifically trained with it in mind (some models are normalized so dot product and cosine become numerically equivalent).
  • Euclidean (L2) distance: less common for text, more standard in other embedding domains.
  • The metric isn't usually a free choice: it should match whatever the embedding model was actually trained with, since that determined what "close" meant during training. Mismatching it silently degrades retrieval quality with no obvious error.

Six databases, one corpus

Every demo below indexes the same 121 chunks from the standard sample paper used throughout this site, each tagged with the real PDF page number it came from — so a metadata filter like "only pages 7–9" or "page ≤ 10" is filtering on something genuine, not a synthetic label. Every demo uses M=32, ef_construction=200, ef_search=64 (or that database's equivalent names) unless noted.

FAISS

A library, not a server or a full database — no persistence, metadata store, or network API of its own; IndexHNSWFlat and IndexIVFFlat are data structures you embed directly in a process. That makes it the fastest option to prototype with and the one with no operational surface at all, at the cost of having to build metadata filtering, persistence, and multi-tenancy yourself if you need them (the IndexIDMap2 + external {id: page} dict pattern below is exactly that DIY layer). Official docs & wiki →

def build_corpus(pdf_path) -> list:
    """Chunks a PDF, tagging each chunk with the real page number it
    came from -- this page number is the metadata every database demo
    below filters on."""
    from pypdf import PdfReader

    CHUNK_SIZE = 400
    CHUNK_OVERLAP = 50

    reader = PdfReader(str(pdf_path))
    chunks = []
    chunk_id = 0
    for page_num, page in enumerate(reader.pages, start=1):
        text = (page.extract_text() or "").strip()
        if not text:
            continue
        for start in range(0, len(text), CHUNK_SIZE - CHUNK_OVERLAP):
            piece = text[start:start + CHUNK_SIZE].strip()
            if len(piece) < 40:
                continue
            chunks.append({"id": chunk_id, "text": piece, "page": page_num})
            chunk_id += 1
    return chunks

def demo_faiss(chunks: list, embeddings, query: str, embed_query) -> None:
    """FAISS has no native metadata -- IndexIDMap2 + an external
    {id: page} dict is the standard pattern, and filtering happens via
    IDSelectorBatch passed as `sel=` in SearchParameters at query time.
    `embed_query(query)` returns a flat 1D float32 vector; FAISS wants
    a 2D batch of query vectors even for a single query."""
    import faiss
    import numpy as np

    dim = embeddings.shape[1]
    page_by_id = {c["id"]: c["page"] for c in chunks}
    ids = np.array([c["id"] for c in chunks], dtype="int64")

    hnsw = faiss.IndexHNSWFlat(dim, 32, faiss.METRIC_INNER_PRODUCT)
    hnsw.hnsw.efConstruction = 200
    hnsw.hnsw.efSearch = 64
    index = faiss.IndexIDMap2(hnsw)
    index.add_with_ids(embeddings, ids)

    query_vec = embed_query(query).reshape(1, -1).astype("float32")

    scores, result_ids = index.search(query_vec, 3)
    print("Unfiltered:", [(page_by_id[cid], round(float(s), 3)) for s, cid in zip(scores[0], result_ids[0])])

    # Metadata filter: restrict the search to a page range using
    # IDSelectorBatch, passed at query time via SearchParametersHNSW.
    allowed_ids = np.array([c["id"] for c in chunks if 7 <= c["page"] <= 9], dtype="int64")
    selector = faiss.IDSelectorBatch(allowed_ids)
    params = faiss.SearchParametersHNSW(sel=selector)
    scores, result_ids = index.search(query_vec, 3, params=params)
    print("Filtered to pages 7-9:", [(page_by_id[cid], round(float(s), 3)) for s, cid in zip(scores[0], result_ids[0]) if cid != -1])

Measured, unfiltered top-3 for "What optimizer and learning rate schedule were used for training?": all three hits land on page 7 (0.592, 0.438, 0.385) — restricting the same search to pages 7–9 returns the identical three results, since they were already inside that range.

Import order matters on macOS

faiss needs to be imported before torch (pulled in by sentence-transformers) in the same process — the reverse order loads two conflicting OpenMP runtimes and segfaults on the first FAISS call.

ChromaDB

A lightweight, developer-friendly database with HNSW as its only local index type (its distributed/cloud tier switches to SPANN internally, currently non-configurable). Metadata is native — a where filter travels in the same call as the vector query — and every HNSW setting lives in one structured configuration={"hnsw": {...}} dict at collection-creation time. space, max_neighbors, and ef_construction are fixed for the life of the collection; ef_search, num_threads, batch_size, sync_threshold, and resize_factor can all be changed later without a rebuild. Official docs →

def demo_chromadb(chunks: list, embeddings, query: str, embed_query) -> None:
    """ChromaDB's metadata is native -- a `where` filter alongside the
    vector query -- and HNSW tuning lives in a structured
    `configuration={"hnsw": ...}` dict passed at collection-creation
    time. Current field name is `max_neighbors`, not the more common `M`."""
    import chromadb

    client = chromadb.EphemeralClient()
    collection = client.create_collection(
        name="attention_paper",
        configuration={"hnsw": {"space": "cosine", "ef_construction": 200, "ef_search": 64, "max_neighbors": 32}},
    )
    collection.add(
        ids=[str(c["id"]) for c in chunks],
        embeddings=embeddings.tolist(),
        metadatas=[{"page": c["page"]} for c in chunks],
        documents=[c["text"] for c in chunks],
    )

    query_vec = [embed_query(query).tolist()]

    result = collection.query(query_embeddings=query_vec, n_results=3)
    print("Unfiltered:", [(m["page"], round(d, 3)) for m, d in zip(result["metadatas"][0], result["distances"][0])])

    # Native metadata filter combined with the vector query in one call.
    result = collection.query(query_embeddings=query_vec, n_results=3, where={"page": {"$in": [8, 9]}})
    print("Filtered to pages [8, 9]:", [(m["page"], round(d, 3)) for m, d in zip(result["metadatas"][0], result["distances"][0])])

Measured, unfiltered top-3 for "What BLEU score did the model achieve on English-to-German translation?": page 8 (0.248), page 1 (0.278), page 8 (0.301) — the abstract on page 1 mentions the same BLEU number in summary form, so it's a genuine near-match, not noise. Filtering to where page in [8, 9] replaces that page-1 result with the next real page-8 chunk (0.342), keeping the result set anchored to where the actual number is derived rather than just restated.

Current field name: max_neighbors, not M

ChromaDB's configuration={"hnsw": {...}} dict uses max_neighbors for what FAISS, Qdrant, and Milvus all call M — passing M raises InvalidArgumentError in the current release.

Qdrant

A full-featured, filter-first database — Qdrant's own architecture applies metadata filters during graph traversal rather than strictly before or after the ANN search, which is part of why it's frequently cited for the strongest filtering performance among open-source options. Below a configurable size (full_scan_threshold, default 10,000 KB) it skips the graph entirely and does an exact scan, since HNSW's overhead isn't worth it for a small enough collection. Embedded mode (QdrantClient(":memory:") or path=...) needs no server at all. Official docs →

def demo_qdrant(chunks: list, embeddings, query: str, embed_query) -> None:
    """`.search()` was removed from the qdrant-client entirely (not just
    deprecated) -- `query_points()` is the only method now. Embedded
    mode (`QdrantClient(":memory:")` or `path=`) needs no server."""
    from qdrant_client import QdrantClient, models

    dim = embeddings.shape[1]
    client = QdrantClient(":memory:")
    client.create_collection(
        collection_name="attention_paper",
        vectors_config=models.VectorParams(
            size=dim,
            distance=models.Distance.COSINE,
            hnsw_config=models.HnswConfigDiff(m=32, ef_construct=200),
        ),
    )
    client.upsert(
        collection_name="attention_paper",
        points=[
            models.PointStruct(id=c["id"], vector=embeddings[i].tolist(), payload={"page": c["page"]})
            for i, c in enumerate(chunks)
        ],
    )

    query_vec = embed_query(query).tolist()

    result = client.query_points(collection_name="attention_paper", query=query_vec, limit=3)
    print("Unfiltered:", [(p.payload["page"], round(p.score, 3)) for p in result.points])

    # Native metadata filter, e.g. excluding appendix pages via a Range
    # condition on the real page-number payload field.
    result = client.query_points(
        collection_name="attention_paper",
        query=query_vec,
        limit=3,
        query_filter=models.Filter(must=[models.FieldCondition(key="page", range=models.Range(lte=10))]),
    )
    print("Filtered to page <= 10:", [(p.payload["page"], round(p.score, 3)) for p in result.points])

Measured, unfiltered top-3 for "How many attention heads does the base model use?": page 15 (0.632), page 9 (0.597), page 7 (0.568) — page 15 is the appendix's attention-visualization figures, a real but tangential match. Filtering to page <= 10 excludes it and surfaces page 5 instead (0.568, tied with page 7), which is the actual model-architecture section defining the head count.

.search() was removed, not just deprecated

qdrant-client v1.16 fully removed .search(), .search_batch(), .recommend(), and several other methods — query_points() is the only search entry point on the current client. Per-query recall tuning goes through search_params={"hnsw_ef": ...} (falls back to ef_construct if unset), and search_params={"exact": True} forces brute-force search — useful for computing a local recall baseline to check ANN parameters against.

Milvus Lite

Milvus is the option built for genuinely large, multi-tenant production deployments, with by far the widest index-type catalog of the six: FLAT, IVF_FLAT, IVF_SQ8, IVF_PQ, HNSW and its quantized variants (HNSW_SQ/PQ/PRQ), DiskANN, GPU-accelerated IVF variants, binary-vector indexes, a sparse-vector index, and AUTOINDEX (which picks parameters for you). Milvus Lite is the same API running embedded and pure-Python, for local development that matches production code one-to-one. Official docs →

def demo_milvus(chunks: list, embeddings, query: str, embed_query, db_path: str) -> None:
    """Milvus Lite is a local, embedded, pure-Python MilvusClient --
    despite the ".db" suffix suggesting a single file, it actually
    creates a directory (WAL + storage), so cleanup needs shutil.rmtree,
    not Path.unlink. Filtering uses a string expression syntax rather
    than a structured filter object."""
    import shutil

    from pymilvus import DataType, MilvusClient

    shutil.rmtree(db_path, ignore_errors=True)
    client = MilvusClient(db_path)

    dim = embeddings.shape[1]
    schema = client.create_schema(auto_id=False)
    schema.add_field("id", DataType.INT64, is_primary=True)
    schema.add_field("page", DataType.INT64)
    schema.add_field("vector", DataType.FLOAT_VECTOR, dim=dim)

    index_params = client.prepare_index_params()
    index_params.add_index(field_name="vector", index_type="HNSW", metric_type="COSINE", params={"M": 32, "efConstruction": 200})

    client.create_collection(collection_name="attention_paper", schema=schema, index_params=index_params)
    client.insert(
        collection_name="attention_paper",
        data=[{"id": c["id"], "page": c["page"], "vector": embeddings[i].tolist()} for i, c in enumerate(chunks)],
    )

    query_vec = embed_query(query).tolist()

    result = client.search(collection_name="attention_paper", data=[query_vec], limit=3, output_fields=["page"], search_params={"params": {"ef": 64}})
    print("Unfiltered:", [(h["entity"]["page"], round(h["distance"], 3)) for h in result[0]])

    # Filter expression syntax -- a real string expression evaluated
    # against the schema field, not a structured filter object.
    result = client.search(
        collection_name="attention_paper", data=[query_vec], limit=3, output_fields=["page"],
        filter="page >= 7 and page <= 9", search_params={"params": {"ef": 64}},
    )
    print("Filtered 'page >= 7 and page <= 9':", [(h["entity"]["page"], round(h["distance"], 3)) for h in result[0]])

Measured results match FAISS exactly (same cosine metric, same embeddings): unfiltered top-3 all on page 7 (0.592, 0.438, 0.385), unchanged after filtering to page >= 7 and page <= 9.

The .db path is a directory, not a file

Despite the .db suffix suggesting a single file, MilvusClient("./file.db") creates a directory (WAL + storage) — clearing it between runs needs shutil.rmtree, not Path.unlink, which raises PermissionError on a directory. Local, serverless use also needs the milvus_lite extra: pip install "pymilvus[milvus_lite]" — a plain pip install pymilvus doesn't pull it in. Milvus's own docs give parameter ranges for HNSW (M: 2–2048, efConstruction: 1–INT_MAX) rather than one fixed default — AUTOINDEX is the intended default path if you don't want to pick values yourself.

pgvector

Not a database of its own — a Postgres extension, so vectors and metadata live as ordinary table columns and filtering is just SQL WHERE combined with ORDER BY ... <=> ... LIMIT, no separate filter API to learn. Supports both HNSW and IVFFlat as index types, plus a feature none of the other five have: iterative index scans (SET hnsw.iterative_scan = strict_order), which keep expanding the search when a highly selective WHERE filter would otherwise starve a fixed-size ANN candidate list down to too few results. Official docs (README) →

def demo_pgvector(chunks: list, embeddings, query: str, embed_query, dsn: str) -> None:
    """Vectors and metadata live as ordinary Postgres columns, so
    filtering is just SQL WHERE combined with ORDER BY ... <=> ... LIMIT
    -- no separate filter API to learn. Needs a running Postgres with
    the pgvector extension, e.g.:
    docker run -d -e POSTGRES_PASSWORD=postgres -p 5433:5432 pgvector/pgvector:pg17
    """
    import psycopg
    from pgvector.psycopg import register_vector

    dim = embeddings.shape[1]
    with psycopg.connect(dsn, autocommit=True) as conn:
        conn.execute("CREATE EXTENSION IF NOT EXISTS vector")
        register_vector(conn)
        conn.execute("DROP TABLE IF EXISTS attention_paper")
        conn.execute(f"CREATE TABLE attention_paper (id BIGINT PRIMARY KEY, page INT, embedding VECTOR({dim}))")

        with conn.cursor() as cur:
            cur.executemany(
                "INSERT INTO attention_paper (id, page, embedding) VALUES (%s, %s, %s)",
                [(c["id"], c["page"], embeddings[i]) for i, c in enumerate(chunks)],
            )

        conn.execute("CREATE INDEX ON attention_paper USING hnsw (embedding vector_cosine_ops) WITH (m = 32, ef_construction = 200)")
        conn.execute("SET hnsw.ef_search = 64")

        query_vec = embed_query(query)

        rows = conn.execute(
            "SELECT page, embedding <=> %s AS distance FROM attention_paper ORDER BY embedding <=> %s LIMIT 3",
            (query_vec, query_vec),
        ).fetchall()
        print("Unfiltered:", [(page, round(d, 3)) for page, d in rows])

        rows = conn.execute(
            "SELECT page, embedding <=> %s AS distance FROM attention_paper WHERE page BETWEEN 8 AND 9 ORDER BY embedding <=> %s LIMIT 3",
            (query_vec, query_vec),
        ).fetchall()
        print("Filtered WHERE page BETWEEN 8 AND 9:", [(page, round(d, 3)) for page, d in rows])

Measured results match ChromaDB exactly, same embeddings and metric: unfiltered top-3 of page 8 (0.248), page 1 (0.278), page 8 (0.301); WHERE page BETWEEN 8 AND 9 produces the same page-8-only result set ChromaDB's where filter did.

Weaviate

The option with the deepest built-in tuning surface of the six: beyond ef/ef_construction/max_connections, Weaviate exposes dynamic_ef_min/dynamic_ef_max/dynamic_ef_factor (dynamic per-query ef, on by default), flat_search_cutoff (falls back to brute-force below 40,000 objects, the same small-collection philosophy as Qdrant's full_scan_threshold), vector_cache_max_objects, and a filter_strategy choice between acorn and sweeping for filtered search. It also ships a genuine standalone flat/brute-force index type (Configure.VectorIndex.flat), recommended specifically for small collections and multi-tenant setups where each tenant's own index is small enough that HNSW's graph overhead isn't worth paying. Vectors are supplied directly (self_provided, since embeddings already exist here) and the HNSW config nests inside Configure.Vectors.self_provided(...) as of client ≥ 4.16 — not a bare top-level kwarg, a common source of bugs against older tutorials. Official docs →

def demo_weaviate(chunks: list, embeddings, query: str, embed_query) -> None:
    """Vectors are supplied directly (self_provided, skipping Weaviate's
    built-in vectorizer modules since embeddings already exist), and
    HNSW config nests inside Configure.Vectors.self_provided(...) as of
    client >= 4.16 -- not a bare top-level kwarg, a common source of
    stale-tutorial bugs. Needs a running server:
    docker run -d -p 8080:8080 -p 50051:50051 -e AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true \\
      -e DEFAULT_VECTORIZER_MODULE=none cr.weaviate.io/semitechnologies/weaviate:1.39.5
    """
    import weaviate
    from weaviate.classes.config import Configure, DataType, Property
    from weaviate.classes.query import Filter

    client = weaviate.connect_to_local()
    try:
        if client.collections.exists("AttentionPaper"):
            client.collections.delete("AttentionPaper")
        collection = client.collections.create(
            name="AttentionPaper",
            properties=[Property(name="page", data_type=DataType.INT)],
            vector_config=Configure.Vectors.self_provided(
                vector_index_config=Configure.VectorIndex.hnsw(ef_construction=200, max_connections=32),
            ),
        )

        with collection.batch.dynamic() as batch:
            for i, c in enumerate(chunks):
                batch.add_object(properties={"page": c["page"]}, vector=embeddings[i].tolist(), uuid=weaviate.util.generate_uuid5(c["id"]))

        query_vec = embed_query(query).tolist()

        result = collection.query.near_vector(near_vector=query_vec, limit=3, return_metadata=["distance"])
        print("Unfiltered:", [(o.properties["page"], round(o.metadata.distance, 3)) for o in result.objects])

        result = collection.query.near_vector(
            near_vector=query_vec, limit=3, return_metadata=["distance"],
            filters=Filter.by_property("page").less_or_equal(10),
        )
        print("Filtered to page <= 10:", [(o.properties["page"], round(o.metadata.distance, 3)) for o in result.objects])
    finally:
        client.close()

Measured results match Qdrant's pattern exactly (same relative distances, cosine-consistent): unfiltered top-3 of page 15 (0.368), page 9 (0.403), page 7 (0.432); filtering to page <= 10 again drops the page-15 appendix hit for page 5.

Pinecone: cloud-only, documented here but not run

Pinecone has no self-hosted path at all, so it isn't part of this page's tested corpus — everything above needed nothing but a local process or a local Docker container. Its current API (the package is now pinecone, not pinecone-client — that name stopped working as of v6.0.0):

from pinecone import Pinecone, ServerlessSpec

pc = Pinecone(api_key="...")
pc.create_index(
    name="attention-paper",
    dimension=384,
    metric="cosine",
    spec=ServerlessSpec(cloud="aws", region="us-east-1"),
)
index = pc.Index("attention-paper")
index.upsert(vectors=[{"id": "0", "values": [...], "metadata": {"page": 7}}])
index.query(vector=[...], filter={"page": {"$gte": 7, "$lte": 9}}, top_k=3, include_metadata=True)

Metadata filtering uses a Mongo-style operator set ($eq, $in, $gte, $and, $or, and others) in the same call as the vector search. For multi-tenant use, Pinecone's own guidance recommends one namespace per tenant by default — noisy-neighbor isolation and a single delete for tenant offboarding — reserving a shared index with metadata filtering for when cross-tenant queries are genuinely required (at the cost of losing per-tenant cost tracking), and a fully separate index per tenant only for compliance-tier isolation needs.

Choosing a vector database is a system design question

The core trade-off is a triangle, not a single axis: recall, latency, and memory — pick any two. Want high recall and low latency? Pay in memory (a denser HNSW graph, or brute force for small enough data). Want low memory and high recall? Pay in latency (smaller, cheaper index, more candidates scanned). Want low memory and low latency? Pay in recall. Quantization is the senior-level fourth lever on top of that triangle: int8 or product quantization commonly buys 4–16× memory reduction for only 1–3% recall loss, which is often a better trade than accepting worse latency or recall directly — worth naming explicitly rather than treating the trade-off as strictly three-way.

The default-then-escalate pattern is what separates a senior answer from a name-drop. Naming Pinecone or Milvus first, before establishing why the workload needs a dedicated system, reads as reaching for a tool rather than reasoning about the problem. The stronger opening: most workloads under roughly 10–50M vectors run cleanly in Postgres with pgvector alongside the primary data — one fewer system, SQL joins between vectors and application data for free — and a dedicated vector database earns its place only once a concrete constraint demands it: scale past what a single Postgres instance handles well, a hard sub-20ms p99 under real concurrent load, or a specific capability (native hybrid search, best-in-class filtering) the simpler option lacks.

Scale bands, synthesized from FAISS's own guidance plus multiple vendors' independent framing (treat the exact cutoffs as commonly-cited ranges, not one universal law):

Scale What actually works Why
Under ~10K–100K vectors Flat / brute-force, no ANN index at all Simpler to build and debug, and fast enough that approximation buys nothing — reaching for a dedicated vector database here is itself a system design mistake
~50K up to 1M–50M HNSW (the default for most workloads) Best recall/latency balance at moderate scale; memory cost is usually still affordable
1M–100M IVF or IVF-PQ becomes competitive with HNSW Lower memory overhead than a graph; Airbnb's own case study explicitly chose IVF over HNSW here for their production constraints — a real counter-example to reflexively assuming HNSW always wins
100M–1B+ Memory becomes the binding constraint, not an implementation detail Uber's own writeup states their 1.5B-item HNSW graph must be fully RAM-resident or latency degrades to "tens of seconds" — this is the regime where PQ compression or a disk-based index (DiskANN, SPANN) stops being optional

One important, honest caveat: not every interviewer is grading vendor knowledge at all. A frequently echoed sentiment from real ML system design interview experiences is that some interviewers explicitly aren't interested in "DB choice, NoSQL trade-offs" and instead focus entirely on embedding serving, the ANN/recall/latency reasoning, and the retrieval funnel — meaning the reasoning about scale → index type → tuning matters more than which product name gets picked.

Real production trade-offs, not hypothetical ones

Notion → turbopuffer: a cost- and multi-tenancy-driven migration at real SaaS scale — 600× growth in daily onboarding, 15× in active workspaces. Moved from a dedicated-hardware "pod" architecture to a serverless, object-storage-backed one (turbopuffer), cutting search infrastructure cost by 60% and improving query latency from 70–100ms down to 50–70ms. Full case study →

Uber → OpenSearch, billion-scale: 1.5B+ items, ~400-dim vectors, targeting 100ms p99 at 2,000 QPS on HNSW. The stated hard constraint — the HNSW graph must be fully RAM-resident or query latency explodes to "tens of seconds" — is the concrete, sourced version of the memory-vs-latency trade-off above. Full case study →

Airbnb: choosing IVF over HNSW: explicitly evaluated both for serving listing-embedding lookups and chose IVF as the better speed/accuracy trade-off for their actual production constraints — useful as a direct counter-example to treating HNSW as unconditionally the right default. Full case study →

Spotify: Annoy → Voyager: a reminder that non-algorithmic constraints can dominate the decision. Voyager (built on hnswlib) benchmarked at 10× faster than their older Annoy library at equal accuracy, using 4× less memory via 8-bit quantization — but the two explicitly named non-performance drivers for the rewrite were statelessness and language support, not benchmark numbers. Full case study →

More real production case studies — GitHub Copilot's embedding model, LinkedIn's GraphRAG deployment, Dropbox Dash's RAG-to-agent evolution, Sourcegraph Cody's decision to abandon embeddings, and others spanning agentic systems, cost optimization, and security incidents — are in the dedicated System Design & Case Studies section.

Follow-up questions interviewers actually probe

These map directly to the real engineering trade-offs above, so even framed generically they're grounded in real system behavior, not trivia:

  • "What happens when the index no longer fits in memory?" — the direct answer is disk-based ANN (DiskANN/SPANN, above), or PQ compression if a moderate recall loss is acceptable.
  • "How do you handle a highly selective filter versus a broad one?" — a filter that only passes a tiny fraction of the corpus (well under 0.1% at million-scale is a commonly cited threshold) favors pre-filtering, since post-filtering a fixed-size ANN candidate list would return too few results. A broad, low-selectivity filter favors post-filtering or Qdrant/pgvector's approach of filtering during graph traversal rather than strictly before or after it.
  • "How would you re-index without downtime?" — shadow indexing (build the new index in the background, promote it once caught up — the pattern Google BigQuery documents for its own vector indexes), or a blue/green cluster swap, the approach Uber used for their own billion-scale reindex.
  • "How do you handle multi-tenant isolation?" — Pinecone's own three-tier framework (above) generalizes across databases: namespace-per-tenant by default, shared index with metadata filtering only when cross-tenant queries are required, dedicated index per tenant only for compliance-tier isolation.
  • "How would this change at 10× scale?" — usually quantization first (a 4–16× memory win for a small recall cost is the highest-leverage single change), then sharding strategy, then whether an in-memory index needs to become disk-based.

A practical decision path

  1. Already on Postgres, and under roughly 10–50M vectors? Use pgvector — one fewer system to operate, and SQL joins between vectors and the rest of the application's data in the same transactional database.
  2. Need zero-ops and cost isn't the binding constraint? Pinecone — fully managed, no infrastructure to run, worth it when developer velocity matters more than infrastructure cost or self-hosting isn't otherwise required.
  3. Past pgvector's comfortable range and need to self-host? Weaviate for hybrid search and filtering built in, Qdrant for the strongest filtering performance and native sparse/hybrid support, Milvus for genuinely billion-scale horizontal production with the widest index-type catalog.
  4. Just prototyping, no production stakes yet? ChromaDB for the fastest local setup, or FAISS specifically when the goal is raw speed as an embedded library with no server at all.

"Open source" doesn't mean "free forever regardless of scale" — Milvus, Weaviate, and Qdrant all also sell a managed cloud tier on top of the free self-hosted software. The real choice is self-hosting (free software, paying for infrastructure and operational time) versus paying someone else to run it — a managed tier of an open-source database, or Pinecone, the one option in this comparison with no self-hosted path at all.

Database Type License Self-hostable? Best for
FAISS Library, not a full DB MIT Yes — just a library, no server Research, prototyping, embedding in an application; build persistence, filtering, and multi-tenancy yourself
Milvus Full-featured DB Apache 2.0 Yes (or Zilliz Cloud) Billion-scale production needing metadata filtering, multi-tenancy, and the widest index-type selection
Weaviate Full-featured DB BSD-3 Yes (or Weaviate Cloud) Native hybrid search and the deepest built-in HNSW tuning surface of the six
Qdrant Full-featured DB Apache 2.0 Yes (or Qdrant Cloud) Strongest filtering performance; native sparse/hybrid support; frequently the most generous free tier
ChromaDB Lightweight, developer-friendly Apache 2.0 Yes Prototyping and quick local development
pgvector PostgreSQL extension PostgreSQL license Yes — it's your own Postgres Teams already on Postgres wanting one fewer system, comfortable up to roughly tens of millions of vectors
Pinecone Managed/hosted Proprietary No — cloud-only Zero-ops teams willing to pay for it

Beyond index choice, three more considerations that come up in both production and interviews: pre- vs. post-filtering (can the database combine a metadata filter with the ANN search itself, or does it filter a fixed result set afterward — the selectivity question above); update and deletion support (some HNSW implementations aren't naturally efficient for high-churn corpora, since removing a node can require partial graph rebuilding); and horizontal scaling (does the system shard across machines as vector count grows into the billions — a concern at Milvus/Uber scale that's irrelevant at FAISS-prototype scale).

Scenario Check

Sources & further reading