Skip to the document
Madhuopen lab

§6 Learn · a course

Thinking in Vectors

From "what is a vector?" to shipping retrieval over a hundred million documents.

66 chaptersabout 15 hours469 recall cardsnot startedBegin chapter 1 →
About this book — who it is for, and how to read it

From "what is a vector?" to shipping retrieval over a hundred million chunks.

This is a book written as a series of blog posts.

There are sixty-six chapters. Most are a 10–15 minute read, and each one can stand alone. They are arranged so that if you start at Chapter 1 knowing nothing and finish at Chapter 66, you will understand modern vector search the way a practitioner does: not as a list of library names, but as a small set of ideas that fit together.

You will not find "and then the magic happens" anywhere in here. Every idea is built from something you already understand. It is explained with an everyday example, and then pushed until the example breaks. Knowing where the example breaks is what separates someone who has read about vectors from someone who can fix a broken search system at 2 a.m.

Who this is for

The novice. You have heard "embeddings" and "vector database" and nodded along. You do not know what a dot product is, or you knew once and it has gone. Start at Chapter 1 and skip nothing. Every term is defined the first time it appears, and every formula comes with plain words and a small worked example you can check with a calculator.

The builder. You have shipped a RAG prototype. It demos well, fails in production, and you do not know why. Start at Chapter 17, read Chapters 40–41 (hybrid search and rerankers), then Part VII (RAG) and Part VIII (Scale). Come back to Parts II and IV when something breaks.

The ninja-in-training. You know HNSW and you have tuned efSearch. You want ColBERT, ColPali, MUVERA, learned sparse retrieval, and the honest economics of billion-scale systems. Read Parts V, VI, VIII and IX.

The cast

This book explains its ideas with five characters and one running example. Meet them once, in Chapter 1, and every later chapter gets easier.

CharacterWhat it really isFirst appears
The Great LibraryYour corpus, every document you want to searchCh. 1
The Map RoomThe embedding space, where meaning becomes locationCh. 1
The Card CatalogThe index, the structure that makes search fastCh. 1
The LibrarianThe retriever, which fetches candidatesCh. 1
The ScholarThe LLM, which reads what the Librarian brings and answersCh. 1

The running example is Acme's support knowledge base: 40,000 pages of help articles, pricing pages, release notes and scanned contracts. We keep asking it one question:

"Does the Pro plan include single sign-on?"

The answer is split across two pages. That is exactly the kind of question that breaks naive retrieval. We watch it fail in many different ways, and fix it one chapter at a time.

Globex, one of Acme's largest customers, first appears in Chapter 33 and returns in many chapters of Parts VI–IX. It has been on Pro with 120 seats since June 2026 (previously Basic with 30), and its scanned contract sits on pages 3,507–3,508.

The whole book, in one sentence: we teach a Librarian to find the right books in a Library too big to read, by giving every book a place in a Map Room and building a Card Catalog over those places, so that the Scholar can answer from the right pages.

How every chapter works

  • "The one-paragraph version" opens every chapter. If that paragraph is all you need, close the tab. No hard feelings.
  • "We will cover the following" comes next, so you always know where you are.
  • The teaching sections follow one rhythm: what the idea is, why we need it, how it works step by step, a worked example where the old way gets the answer wrong and the new way gets it right, and when to use it.
  • "Under the hood" has the code. It is Python with NumPy, written to be read, not pasted. Where a real library matters, it is named.
  • "What people get wrong" exists because the fastest way to learn a field is to inherit other people's scar tissue.
  • "Ninja notes" are the parts that only matter once you have real traffic. Novices can skip them on the first pass and should come back on the second.
  • "Key takeaways" and "What's next" close every chapter. Read only those and you still get a correct, shallow map of the whole book.
  • Maths appears, but never without a plain-English translation and a small worked example right beside it. Skip every equation and you will still finish the book understanding vector search. The equations are there to make that understanding precise.

Note: Two chapters skip parts of this template on purpose. Chapter 65 is an essay about where the field is heading. Chapter 66 is a reference: the book's key decision trees, formulas and defaults in one place.

Part I — Foundations: What a Vector Actually Is

01What Is a Vector, Really?A vector is a list of numbers. It is also a place. Once you see both at once, everything else in this book follows.10 min02From Things to Numbers: The Leap of EmbeddingAn embedding is a learned function from a thing to a place. Here is what 'learned' actually buys you, and what it quietly costs.11 min03The Geometry of MeaningA tour of the Map Room: neighbourhoods, directions, clusters, and the famous king − man + woman trick — including why it works less well than you have been told.11 min04Measuring Similarity: Dot, Cosine, EuclideanThree rulers for the Map Room, what each one actually measures, when they agree, and the specific ways each one will mislead you.11 min05Normalization and the Unit SphereDividing a vector by its own length is a one-line operation with outsized consequences. Here is what it buys, what it costs, and when to skip it.9 min06High Dimensions Are a Strange CountryIn 768 dimensions, almost everything is far away, almost everything is orthogonal, and distance stops being informative. Vector search works anyway — and the reason why is the most important idea in Part I.11 min07Dense vs Sparse VectorsTwo completely different shapes of vector, two completely different index structures, two different failure modes — and the reason serious systems run both.11 min

Part II — Where Embeddings Come From

08Before Embeddings: One-Hot, Bag-of-Words, TF-IDF, BM25The counting-based ancestors of every modern retrieval system — and why BM25, a formula from 1994, is still the baseline you must beat.14 min09Word2Vec and the Distributional HypothesisOne idea — you shall know a word by the company it keeps — turned into the training trick that made dense embeddings practical, and that every modern model still uses.12 min10Transformers and Contextual EmbeddingsAttention lets every token's vector be rewritten by its neighbours. That single change is why 'bank' finally has two meanings — and why nearly every embedding model since 2018 is a transformer.11 min11From Tokens to One Vector: PoolingA transformer hands you 512 vectors. You need one. The way you collapse them is a real design decision — and the road not taken here becomes ColBERT.12 min12Contrastive Learning: How Embedding Models Are TrainedPull the pair together, push the crowd apart. The objective is four lines of code. Everything that makes a model good or bad lives in how you choose the negatives.15 min13Bi-Encoders vs Cross-EncodersEncode separately and be fast, or read together and be right. This single trade-off is the skeleton of every serious retrieval pipeline — and late interaction is the compromise between them.15 min14Asymmetric Search and Instruction-Tuned EmbeddingsA question is not a short answer. Modern embedding models want a prefix telling them which side you are on — and forgetting it is one of the most common silent bugs in RAG.11 min15Matryoshka EmbeddingsTrain a model so the first 64 dimensions are already a good embedding, the first 256 are better, and all 768 are best. Then truncation becomes a free tuning knob.11 min16Choosing and Benchmarking an Embedding ModelLeaderboards measure someone else's problem. Here is the decision framework, the seven criteria that actually matter, and the one-day evaluation that beats any ranking.13 min

Part III — Finding Neighbours

17Brute Force and the Flat IndexComparing the query to every single vector is exact, trivial to implement, and far faster than most people believe. Know exactly when you can get away with it.13 min18The Recall–Latency–Memory TriangleEvery ANN index is a position on a three-way trade-off. Once you can name the three axes, the entire zoo of algorithms in Part IV becomes a single map.13 min19Measuring Retrieval QualityRecall@k, MRR, nDCG, and the crucial distinction between system recall and index recall — two different numbers that share a name and get confused constantly.14 min20The ANN Family TreeEvery approximate nearest neighbour algorithm ever built is one of four ideas, or a combination of them. Learn the four and the zoo becomes a map.11 min

Part IV — Index Structures

21LSH: Hashing Things That Are AlikeA hash function designed to collide. Rarely the right choice for dense retrieval today, but the ideas inside it reappear in binary quantization and in MUVERA.12 min22Trees: KD-Trees and AnnoySplitting space in half repeatedly is the textbook answer to nearest-neighbour search. It is also the clearest demonstration of what the curse of dimensionality does to an algorithm.10 min23IVF: Clustering the SpaceGroup vectors into clusters, compare the query to the cluster centres, and search only the closest few. One idea, one knob, and the foundation of most billion-scale systems.14 min24Product QuantizationChop a vector into pieces, replace each piece with the nearest entry from a small codebook, and store only the codebook IDs. 3 KB becomes 96 bytes and search still works.15 min25IVF-PQ and OPQ in PracticeCombine clustering with quantization and you get the index that serves most of the world's billion-vector workloads. Here is how to configure and tune it without guessing.12 min26Scalar and Binary QuantizationTwo compression schemes with no codebooks and no training. int8 gives 4× for almost nothing. One bit per dimension gives 32× and, with rescoring, loses surprisingly little.12 min27TurboQuant: Near-Optimal Quantization Without CodebooksScalar quantization is cheap but wasteful. Product quantization is efficient but needs trained codebooks. TurboQuant rotates the problem away and gets near-optimal compression with no training at all.15 min28HNSW I: Small Worlds and Skip ListsSix degrees of separation is not a coincidence — it is a property you can engineer. Combine it with the express-lane trick from skip lists and you get one of the best in-memory vector indexes we know how to build.11 min29HNSW II: Building the GraphM, efConstruction, and the neighbour-selection heuristic that decides whether your graph is navigable or merely connected. This is where HNSW is won or lost.12 min30HNSW III: Searching and TuningBeam search with one runtime knob. How efSearch works, how to read your recall curve, and how to choose a point on it deliberately rather than by superstition.14 min31HNSW in ProductionThe algorithm is elegant. Operating it is not. Deletes that aren't deletes, graphs that degrade, memory that doesn't fit, and the rebuild you will eventually schedule.11 min32DiskANN: Billion-Scale on SSDKeep compressed vectors in RAM to decide where to walk, and read full vectors from SSD only when you arrive. A billion vectors on one machine, at single-digit milliseconds.14 min33Filtered Vector SearchAdding WHERE tenant_id = 42 to a vector search sounds trivial and is the hardest easy problem in the field. Three strategies, each broken in a different regime.13 min

Part V — Beyond One Vector

34The Single-Vector Bottleneck768 numbers cannot hold everything a page contains. Here is precisely what gets lost, why part of it is a geometric limit rather than a model failure, and what the alternatives are.14 min35ColBERT I: Late InteractionWhat is ColBERT? Keep one vector per token instead of one per document, and compare them at the last moment. The result recovers most of a cross-encoder's accuracy while staying fully precomputable.13 min36ColBERT II: MaxSim and TrainingWhy max and not mean, what the scoring function is really computing, why it is asymmetric, and how ColBERT is trained. The details that decide whether it works.15 min37ColBERT III: ColBERTv2, PLAID, and ServingTwo hundred vectors per passage sounds unaffordable. Residual compression cuts storage 14–25x, and PLAID's centroid-based search means full MaxSim runs on a few hundred to a thousand passages out of ten million. Here is how a multi-vector index actually runs.13 min38MUVERA: Multi-Vector Made SingleWhat if you could squash 200 token vectors into one fixed-length vector whose dot product approximates MaxSim? Then every index in Part IV works again, unchanged. That is MUVERA.14 min39SPLADE and Learned Sparse RetrievalA transformer that outputs term weights instead of a dense vector — including weights for words the document never contained. Neural semantics, classical inverted index.12 min40Hybrid Search and FusionRun keyword search and vector search, then merge the two ranked lists. A few lines of code, no training, and the cheapest, most reliable recall win in this book.11 min41RerankersThe last stage before the LLM, and the highest-precision component in the pipeline. Cross-encoders, LLM rerankers, and how to decide how deep to go.12 min

Part VI — Vectors Beyond Text

42Multimodal Embeddings: CLIP and the Shared SpaceTrain two encoders — one for images, one for text — to place matching pairs at the same point. Suddenly you can search photographs with sentences.13 min43SigLIP and Modern Vision-Language EncodersReplace the softmax with a sigmoid and the batch-size tyranny disappears. Plus the other changes that turned vision encoders into something that can read a page.13 min44The Broken Promise of OCR PipelinesOCR, layout, tables, reading order, chunking, embedding. Six stages, six places to lose information, and errors that compound silently. Here is exactly what you lose.13 min45ColPali I: Documents as ImagesWhat is ColPali? Delete the entire parsing pipeline. Take a screenshot of the page, embed the pixels with a vision-language model, and search that. It works better than the pipeline it replaces.12 min46ColPali II: Inside the ModelPatches, projection, and late interaction over pixels. How a page becomes a thousand vectors, how the model was trained, and how to read the similarity heatmaps it gives you for free.12 min47ColPali III: Production Reality527 KB per page times a million pages is 527 GB. Here is the full cost model, the four compression levers, and an honest account of when not to use this.14 min48Audio, Video, Code, Graphs, and UsersA tour of everything else that has been mapped into vector space — and the single recipe that keeps working across all of them.13 min

Part VII — RAG

49RAG From First PrinciplesWhy retrieve at all when context windows are enormous? Because attention is not retrieval, tokens cost money, and your data changes. The honest case for RAG, and its exact shape.13 min50ChunkingHow you cut documents into pieces can matter more for retrieval quality than which embedding model you pick. The trade-off, the strategies, and the techniques that escape it.14 min51Metadata and StructureA large share of retrieval quality comes from things that are not vectors at all: titles, dates, permissions, document type, and the links between documents.11 min52Query UnderstandingUsers write short, ambiguous, context-dependent questions. Rewriting, decomposition, HyDE and multi-query turn them into something a retriever can actually use.11 min53Context AssemblyRetrieval found the right material. Now decide how much of it to send, in what order, with what labels, and how to stop the Scholar answering from its own memory instead.11 min54Evaluating RAGEvaluate retrieval and generation separately, or you will never know which one broke. The four measurements that matter, how to build a golden set in a day, and how far to trust an LLM judge.11 min55RAG Failure Modes: A Field GuideTwelve recurring ways RAG systems break, organised by symptom so you can diagnose from what you observe — plus the cause, the test, and the fix for each.16 min56Agentic RetrievalStop retrieving once, up front. Give the model a search tool and let it decide when to search, what to search for, and whether to search again after reading.10 min

Part VIII — Scale and Production

57Architecture for Millions of DocumentsThe reference design for RAG over millions of documents: separate ingestion from serving, keep a source of truth outside the index, and treat every index as rebuildable.10 min58Sharding, Replication, and RoutingHow one index becomes forty. Random versus semantic sharding, scatter-gather and its tail latency, replication for throughput, and routing that avoids asking every shard.11 min59Freshness and Index MaintenanceDocuments change every minute, and indexes are expensive to modify. The log-structured pattern that reconciles them — deltas, tombstones, compaction — and the maintenance calendar that keeps it healthy.11 min60Cost EngineeringThe arithmetic of vector search at scale — bytes, GPU-hours, and tokens — worked through for a hundred million chunks, with the levers ranked by how much each one saves.11 min61Image-Heavy RAG, End to EndA complete, costed design for question answering over a million scanned, chart-filled, table-heavy pages — assembled from ColQwen, MUVERA, hybrid search and a vision LLM.15 min62Multi-Tenancy, Privacy, and Embedding InversionEmbeddings are not anonymised data — text can be substantially reconstructed from them. Treat vectors like the documents they came from, enforce tenancy before retrieval, and treat retrieved text as untrusted input.12 min63Model Migration and DriftVectors from different models live in different spaces, so changing models means re-embedding everything. How to do it without downtime, how to know when you should, and how to detect drift before users do.13 min

Part IX — Ninja Tier

64Vectors as Agent MemoryEmbedding every message and retrieving the nearest ones is the obvious way to give an agent memory, and it fails in predictable ways. What memory actually requires: extraction, consolidation, recency, and forgetting.12 min65The FrontierWhere vector search appears to be heading, separated into trends with strong evidence, open questions worth watching, and the principles that will outlast all of them.8 min66The Ninja's Field ManualThe book's key decision trees, formulas, defaults and checklists, in one place, each with a pointer back to the chapter that explains it.Reference — keep open while building