Everything you have, running at once
A retrieval system's recall is not its retriever's recall. On the run below the retriever finds every relevant document, and the finished answer is missing one of them, because a later stage threw it away.
Chapter 8 built four instruments and used each one on its own. This chapter runs them in order, on one query, against twelve documents, and prints every number. Think of a sorting line with stations along it. A station can reject an item and a station can change the order of the items, and no station anywhere on the line can hand back something an earlier one dropped.
The twelve documents are positions in the same plane we have used since chapter 1, so every score on this page is arithmetic you can check. The query is best way to make tea at home, embedded at $[5, 1]$.
| Id | Position | Text |
|---|---|---|
| d01 | $[6, 0]$ | how to brew loose leaf tea |
| d02 | $[5, 1]$ | brewing loose-leaf tea at home |
| d03 | $[5, 0]$ | espresso grind size |
| d04 | $[4, 1]$ | a warm drink before bed |
| d05 | $[3, 1]$ | milk for a kitten |
| d06 | $[2, 3]$ | feeding a rescue kitten |
| d07 | $[1, 4]$ | walking a dog in the rain |
| d08 | $[1, 5]$ | dog leash training |
| d09 | $[0, 4]$ | horse pasture rotation |
| d10 | $[2, 2]$ | the farm cafe menu |
| d11 | $[6, 1]$ | iced tea, two ways |
| d12 | $[9, 9]$ | quarterly revenue by region |
Three of them are the right answers: d01, d02 and d11 all tell you how to make tea. I labeled that set by reading the twelve texts, before running anything, and it does not change for the rest of the chapter. A relevant set chosen after seeing the output would let the output decide what counts as right.
Stage one: retrieve
Score all twelve by cosine to the query and keep the best five. This is chapter 8's nearest-neighbor search, unchanged.
$$\cos(\mathbf{q},\, \mathbf{d}) = \frac{\mathbf{q} \cdot \mathbf{d}}{\|\mathbf{q}\|\,\|\mathbf{d}\|}$$In words: the cosine between the query $\mathbf{q}$ and a document $\mathbf{d}$ is their dot product divided by both of their lengths. Dividing by the lengths throws the lengths away, so the ranking is about heading alone.
The twelve cosines, best first: d02 1.000, d11 0.999, d04 0.999, d05 0.992, d01 0.981, d03 0.981, d10 0.832, d12 0.832, d06 0.707, d07 0.428, d08 0.385, d09 0.196. The cut at five keeps d02, d11, d04, d05 and d01.
That d12 score is chapter 2 arriving with consequences. The revenue report sits at $[9, 9]$, far from everything, and cosine ignores how far. It has the same heading as d10, the cafe menu, so it gets the same score. A system that ranked by distance instead would put it last.
Stage two: dedupe, and what it cost
Near-duplicate documents waste the user's attention and the reranker's budget, so most pipelines drop any candidate that is nearly identical to a candidate already kept. Walk the five in rank order and drop anything whose cosine to an already-kept document is 0.999 or higher.
Two pairs clear that bar. cos(d02, d11) is 0.999 and cos(d02, d04) is 0.999. Walking in order keeps d02, then drops d11, then keeps d04 because d04 was never compared against a keeper that scored high enough, then keeps d05 and d01. Four survive.
d11 is iced tea, two ways. It is not a duplicate of anything. It is one of the three documents I labeled relevant before the run, and the threshold killed it.
Stage three: rerank
The four survivors go to a scorer that reads the query and the document together, which chapter 14 calls a cross-encoder. Ours is a toy with two terms: start at three, subtract a sixth of the gap in each coordinate, and add one if the text says "tea".
The scores come out d02 4.000, d01 3.667, d04 2.833, d05 2.667. The order arriving was d02, d04, d05, d01. The order leaving is d02, d01, d04, d05.
d01 is how to brew loose leaf tea, and it was fourth of four on the cosine ranking. It says "tea" and it is close to the query, so the second scorer, which gets to look at both texts at once, puts it second. That is the whole case for reranking, and chapter 14 prices it.
Stage four: score the run
Two numbers score a ranked list against the relevant set, and they differ in their denominators, which is the whole reason they disagree.
$$\text{recall@}k = \frac{|R \cap \text{top-}k|}{|R|} \qquad\qquad \text{precision@}k = \frac{|R \cap \text{top-}k|}{k}$$In words: both count how many of the relevant documents $R$ appear in the first $k$ results. Recall divides that count by how many relevant documents exist, so it asks what share of the right answers you found. Precision divides it by $k$, so it asks what share of what you returned was right. Different denominators, different questions.
| $k$ | Relevant found | recall@$k$ | precision@$k$ |
|---|---|---|---|
| 1 | d02 | 0.333 | 1.000 |
| 3 | d02, d11 | 0.667 | 0.667 |
| 5 | d02, d11, d01 | 1.000 | 0.600 |
| 10 | d02, d11, d01 | 1.000 | 0.300 |
In the third row, recall@5 is 1.000: the retriever found all three. The pipeline handed back d02, d01, d04 and d05, which contains two of the three, so the pipeline's recall is 0.667. The two numbers are measured at different points on the line.
| Station | In | Out | What it can lose |
|---|---|---|---|
| Embed the query | 12 | 12 | Nothing |
| Retrieve top five | 12 | 5 | Seven documents, by the depth you chose |
| Dedupe | 5 | 4 | One document, by the threshold you chose |
| Rerank | 4 | 4 | Nothing; it reorders |
| Score the run | 4 | 4 | Nothing; it measures |
| Cluster | 12 | 12 | Nothing; it groups |
| Flag anomalies | 12 | 11 | One document, if you act on the flag |
| Watch drift | 12 | 12 | Nothing; it compares |
Stage five: cluster
Clustering is not part of answering the query. It is how you find out what is in the collection you are serving. Run chapter 8's k-means on all twelve documents with $k = 3$, starting the centers at $[6, 0]$, $[1, 5]$ and $[2, 2]$.
The within-cluster sum of squares, which is the total of every document's squared distance to its own center, falls at each pass: 76.417, then 70.417, then 20.333, then 16.167, and the fourth pass is where it stops moving. The centers settle at $[4.833, 0.667]$, $[9, 9]$ and $[1.2, 3.6]$.
The groups are the six drink documents, the five animal-and-farm documents, and d12 by itself. One cluster is the revenue report and nothing else. That is the outlier showing up as a group of one, which is a useful thing for a cluster report to tell you and a confusing thing if you expected three topics.
Press Step four times and the readout walks down the same four numbers the paragraph above lists. Then set the clusters to two and watch the revenue report get folded into a real group, which is the same document looking like a member instead of an event.
Stage six: watch it for anomalies
An anomaly here means a document far from the middle of the collection. Take the centroid of all twelve, which sits at $[3.667, 2.583]$, measure every document's distance to it, and you get a mean of 3.083 and a standard deviation of 1.769.
Flag anything past the mean plus two standard deviations, which is 6.621. Two is a convention, not a law, and twelve points do not earn a bell curve; what makes the rule safe on this corpus is the gap that follows. Exactly one document clears it: d12 at 8.344. The next-highest is d09 at 3.931, so there is a wide gap between the flagged document and everything else, and on this corpus the monitor is right. The revenue report does not belong in a recipe collection.
The threshold is also a maintenance job rather than a setting. It was computed from this month's twelve documents. Add a thousand documents about revenue and the mean moves out, the standard deviation grows, and 6.621 stops flagging anything.
Stage seven: watch it for drift
Re-embed the corpus next month. Every document shifts by $[0.3, -0.2]$, which is the kind of small global movement a retrained model produces, and one document genuinely moves: d06, feeding a rescue kitten, ends up at $[2, 5]$.
The corpus centroid goes from $[3.667, 2.583]$ to $[3.942, 2.567]$. The cosine between the old centroid and the new one is 0.999 and the gap between them is 0.276. A monitor watching the corpus-level number reports that nothing happened.
Now measure each document against its own old position. The four lowest cosines are d06 at 0.979, d10 at 0.993, d07 at 0.996 and d05 at 0.996. The event is right there, and it has a name attached to it.
The corpus average hid the one document that moved. Watch both numbers: the corpus-level one tells you whether the whole space moved, and the per-document one tells you which items moved inside it. Chapter 8 introduced drift on a single word; this is what it looks like when you have to run it as a monitor on a collection.
Stage eight: the budget
One query, end to end, on a corpus of a million documents rather than twelve, costs this. The stage counts stay the same; only the search step changes, because a million documents need chapter 8's approximate index.
$$t_{\text{query}} = t_{\text{encode}} + t_{\text{search}} + t_{\text{dedupe}} + t_{\text{rerank}} + t_{\text{assemble}}$$In words: the time for one query is the sum of the five stages, and nothing overlaps, so the total is an addition rather than a maximum. Encoding runs once, the search runs once, and the rerank cost is the number of survivors multiplied by the cost of scoring one pair.
Encoding the query takes 8.000 ms, or 19.465 percent. The approximate search over 1,000,000 documents takes 12.000 ms, or 29.197 percent. Deduping five candidates takes 0.100 ms, or 0.243 percent. Reranking four survivors takes 20.000 ms, or 48.662 percent. Assembling the answer takes 1.000 ms, or 2.433 percent. The total is 41.100 ms.
So the first trade you would make is the rerank depth, because it is the only line that scales with a number you chose. Halve the survivors and you halve the largest segment. Chapter 14 works out what you give up: the rerank is the stage that fixed the ordering, and d01 was the document it rescued.
What this looks like at ten million
Nothing on this page changes shape when the corpus grows. The retrieve stage stops scanning and walks chapter 8's layered graph instead, which returns most of the true neighbors rather than all of them, so the recall you lose at stage one is now a setting in the index as well as a depth you chose. Storage is chapter 14's table: at 384 dimensions a million documents is 1.536 GB in float32 and 0.768 GB in float16, so ten million is ten times that, and it has to sit in memory for the search to be fast. The rerank stays at 5.000 ms per pair no matter how large the corpus gets, because it only ever sees the survivors. That is why the industry shape is a cheap wide first stage and an expensive narrow second one, and it is the same five lines of budget with bigger numbers in two of them.
Where you'll meet this
What you now know
- A pipeline is a sorting line: a station can reject a document or reorder the survivors, and no station can recover something an earlier one dropped.
- Ranking twelve documents by cosine puts a revenue report at 0.832, level with a cafe menu, because cosine compares headings and throws the lengths away.
- A dedupe threshold of 0.999 removes d11, a relevant document, which is what a fixed cutoff on a score does when the score was not built to answer the question you asked it.
- Reranking with a scorer that reads both texts changed three of the four positions and lifted d01 past two documents the cosine had ranked above it.
- recall@5 on the retriever's list was 1.000 while the finished pipeline returned 0.667 of the relevant set, so a pipeline's recall is not its retriever's recall.
- A corpus-level drift cosine of 0.999 said nothing had happened in a month when d06 had moved to 0.979, so the average hides the event the monitor exists to find.
- One query costs 41.100 ms across five stages and the rerank is 48.662 percent of it, so the first thing you trade is how many survivors you rerank.
Where we're headed
Every number in this book so far has been one I could compute in front of you, which was the point. The questions that are actually open in this field cannot be settled that way, and some of them are open because nobody has run the right experiment yet. The last chapter names those questions in the terms the rest of the book has earned, and then hands you three experiments small enough to run on a laptop tonight, each with the control that turns a picture into evidence.