Field notes — AI infrastructure

The Vector Database Illusion

What nobody tells you about RAG, scale, and semantic search.

By Muhammad Zubair Ilyas · 12 min read

Stop treating vector databases like a magic bullet.

I've spent the last stretch building and shipping Retrieval-Augmented Generation (RAG) assistants for a handful of enterprise clients, and here's the thing the marketing decks leave out: your choice of vector database mostly affects speed. Your embedding strategy is what decides whether the thing actually works.

Traditional SQL databases are mathematically blind to meaning. They index keywords, not concepts. Point one at a raw video, a customer support call transcript, or a 200-page 10-K filing, and it falls apart.

So we reach for a vector database. Fair enough — but before you write a single line of infrastructure code, it's worth understanding what's actually happening under the hood: where the hidden costs are buried, and which engine is actually worth your budget.

01 — The Geometry of Meaning

At its core, a vector database doesn't store text — it stores coordinates in a high-dimensional mathematical space.

Machine learning models (OpenAI's text-embedding-3-large, or an open-source Hugging Face model) compress unstructured data into a long list of floating-point numbers called an embedding. A sentence like "The CEO resigned abruptly" becomes a 1,536-dimensional coordinate. A different sentence — "The executive quit suddenly" — lands on a coordinate physically close to the first one, because the two share semantic meaning even though they don't share a single word.

Semantically related concepts cluster together in vector space; the database's job is measuring the distance between them.

The database's core job is calculating the geometric distance between your query coordinate and millions of stored coordinates, using one of three "distance rulers":

• Cosine similarity — measures the angle between two vectors and ignores their length. It cares about direction, not magnitude, which makes it a common default for comparing document topics.

• Euclidean distance (L2) — measures the straight-line distance between two points. Useful when magnitude itself carries information, such as image embeddings where brightness or intensity matters.

• Dot product — combines magnitude and direction. It's the workhorse behind most recommendation engines, since it captures both what a user likes and how strongly they like it. (Worth knowing: on normalized vectors, dot product and cosine similarity are mathematically equivalent — the "gold standard" framing you see in a lot of NLP writing is really about normalization, not the metric itself.)

02 — The 10ms Speed Trick — and Its Hidden Cost

Here's the part that doesn't make it into the intro tutorials: exact nearest-neighbor search (k-NN) basically doesn't scale. Calculate the true distance from a query to every stored vector across a 10-million-row, 768-dimension collection, and you're looking at somewhere in the hundreds of milliseconds to over a second on typical hardware — enough to visibly stall a user-facing product.

So in practice, every production vector database trades a small amount of accuracy for a large amount of speed, using an Approximate Nearest Neighbor (ANN) algorithm instead of an exact one.

Hierarchical Navigable Small World (HNSW)

Picture an express highway system: the top layer is made of sparse, long-distance jumps between "hub" points, and each layer below gets denser and more local. A search starts on the highway to find the right neighborhood fast, then drops down through the layers to pinpoint the exact match.

HNSW's layered graph: broad jumps at the top narrow into local search near the bottom.

The catch: HNSW is memory-hungry. Budget for a RAM footprint noticeably larger than your raw vector size — often 1.5–2x, depending on the graph's connectivity parameters.

Inverted File Index (IVF)

Think of this as a library index instead of a highway map. IVF partitions the vector space into clusters ahead of time, finds the cluster closest to your query, then only searches inside that cluster.

The catch: it's far lighter on RAM, but recall tends to be a bit lower and index builds are more computationally expensive up front.

On public 1-million-vector, 768-dimension benchmarks (see ann-benchmarks.com for a live comparison), HNSW typically lands around 95% recall in the low double-digit milliseconds, while IVF trades a few points of recall for a faster query. Treat these as ballpark figures, not a spec sheet — your actual numbers will move with hardware, index parameters, and dataset shape, so benchmark on your own data before committing.

Rule of thumb: reach for HNSW when recall is non-negotiable. Reach for IVF when you're RAM- or cost-constrained and can tolerate a small accuracy hit.

03 — RAG: Where Theory Meets a Brick Wall

RAG is the vector database's flagship use case — it patches the LLM's frozen training data and cuts down on hallucination. But the textbook RAG diagram hides most of the real engineering work.

The full pipeline: chunking, embedding, storage, re-ranking, then generation. Most of the engineering effort lives in the first and fourth steps.

Pipeline: Chunking → Embedding → Vector DB → Re-ranking → LLM

The chunking nightmare

Splitting documents into chunks sounds trivial until you try it. Fixed-size chunking — say, exactly 512 tokens per chunk — will sooner or later slice straight through a sentence that mattered, destroying its context. Semantic chunking (using a model to detect natural logical breaks) preserves meaning much better, but it typically costs several times more in compute.

Production fix: if you're using fixed-size chunking, overlap adjacent chunks by roughly 15–20% so you don't lose context at the seams.

The "lost in the middle" problem

Even with a clean retrieval step, LLMs show a U-shaped attention pattern: they pay close attention to the start and end of a prompt and tend to skim past the middle. Feed a model ten retrieved chunks, and chunks four through seven are the ones most likely to get ignored.

Production fix — re-ranking:

1. Retrieve the top 20 vector-similar chunks.

2. Score them against the query with a lightweight local cross-encoder (e.g. bge-reranker-large).

3. Pass only the top 5 re-ranked chunks to the LLM.

This adds roughly 200ms of latency, but in my experience it's consistently worth it — the accuracy gain is large enough to notice immediately in production.

The metadata filtering trap

You rarely search on vectors alone in production. It's almost always a vector search combined with a metadata filter: "find Q4 earnings documents where the author is Alice and the date is after 2025." How you combine the two matters more than people expect.

Pre-filtering can shrink your candidate pool to nothing before the vector search even runs; post-filtering searches broadly first, then filters.

• Pre-filtering — apply the metadata filter first, then vector-search the remaining subset. Fast, but if the filter eliminates most of your dataset, your search pool can shrink to the point of being semantically empty.

• Post-filtering — vector-search the whole dataset first, then filter the top results by metadata. Preserves semantic breadth, but you risk zero results if none of your top matches happen to satisfy the filter.

Use pre-filtering when the metadata condition eliminates the large majority of your dataset (e.g. "last 24 hours only"). Use post-filtering when you need maximum semantic breadth. In practice, this single decision tends to move latency more than your choice of ANN index — test both on real traffic before you commit.

04 — The Database Decision Matrix

The market has settled into a few distinct paradigms. Here's how I'd frame the trade-offs:

Four paradigms, four very different operating costs.

05 — Actionable Implementation Blueprint

Here's a minimal, working template — from chunking through running a semantic search against pgvector.

from sentence_transformers import SentenceTransformer
import psycopg2

# 1. Initialize a fast, local embedding model
model = SentenceTransformer('all-MiniLM-L6-v2')

# 2. Chunk text with an overlapping window to preserve context
def chunk_text(text, chunk_size=500, overlap=100):
    chunks = []
    step = chunk_size - overlap
    for i in range(0, len(text), step):
        chunks.append(text[i:i + chunk_size])
    return chunks

# 3. Process a raw document
raw_document = "The executive board met last Thursday to discuss fiscal projections..."
document_chunks = chunk_text(raw_document)

# 4. Generate embeddings
embeddings = model.encode(document_chunks)

# --- Insert into pgvector ---
# Assumes: CREATE TABLE docs (id SERIAL, content TEXT, embedding vector(384));
conn = psycopg2.connect("dbname=vectors user=postgres password=yourpassword host=localhost")
cursor = conn.cursor()

for chunk, vector in zip(document_chunks, embeddings):
    cursor.execute(
        "INSERT INTO docs (content, embedding) VALUES (%s, %s)",
        (chunk, vector.tolist())
    )
conn.commit()

# --- Semantic search ---
user_query = "What did the board decide about finances?"
query_vector = model.encode(user_query)

# pgvector's `<=>` operator returns cosine distance; 1 - distance = similarity
cursor.execute(
    """SELECT content, 1 - (embedding <=> %s) AS similarity
       FROM docs ORDER BY embedding <=> %s LIMIT 5""",
    (query_vector.tolist(), query_vector.tolist())
)
top_chunks = cursor.fetchall()

for chunk in top_chunks:
    print(f"Score: {chunk[1]:.4f} -> {chunk[0][:100]}...")

The #1 hidden migration tax: don't change your embedding model mid-project. Move from text-embedding-ada-002 to text-embedding-3-large and the entire coordinate system shifts underneath you — every existing vector becomes incomparable to new ones. You'll need to re-embed and re-index the whole database from scratch. Budget for that downtime before you start, not after.

Final Verdict: The Semantic Shift

We're moving from an era of exact-match keywords to one of conceptual proximity. Treating data as points in a mathematical space is one of the more significant shifts in how we build data systems since the relational model itself.

But keep the two pieces in proportion: the vector database is the engine, not the whole car. Your data pipeline — chunking, metadata design, re-ranking — is the fuel.

If I had to split my time again, I'd spend roughly 70% of it on the pipeline and 30% on picking the database. Get that balance right and the system feels close to magic. Get it backwards, and you've built an expensive, over-engineered synonym finder.

Muhammad Zubair Ilyas is a software engineering student and full-stack developer.