Part IV · Ch. 15 — The Instruments at Work

Part IV · Chapter 15 of 16

The Instruments at Work

One query, one pipeline, every number printed


Everything you have, running at once

A retrieval system's recall is not its retriever's recall. On the run below the retriever finds every relevant document, and the finished answer is missing one of them, because a later stage threw it away.

Chapter 8 built four instruments and used each one on its own. This chapter runs them in order, on one query, against twelve documents, and prints every number. Think of a sorting line with stations along it. A station can reject an item and a station can change the order of the items, and no station anywhere on the line can hand back something an earlier one dropped.

Eight rounded boxes in a row. The first five, under the heading 'the answer path', are embed the query, retrieve top five, dedupe, rerank and score the run, joined left to right by small arrows. Above each box is a count of documents entering, below it a count leaving, and under that what the station can lose. The counts read 12 in and 12 out, 12 in and 5 out, 5 in and 4 out, 4 in and 4 out, 4 in and 4 out. The retrieve and dedupe boxes are outlined in amber and their out-counts are amber. A dashed vertical line separates the last three boxes, headed 'monitors, reading the whole corpus': cluster, flag anomalies and watch drift, each taking 12 in, with flag anomalies outlined in amber and reading 11 out.
Eight stations, and the count only falls. Along the answer path the running total goes 12, 12, 5, 4, 4. Two stations subtract; none of them adds.

The twelve documents are positions in the same plane we have used since chapter 1, so every score on this page is arithmetic you can check. The query is best way to make tea at home, embedded at $[5, 1]$.

The twelve-document corpus. The positions are the input to every stage below; nothing else about these documents is used.
IdPositionText
d01$[6, 0]$how to brew loose leaf tea
d02$[5, 1]$brewing loose-leaf tea at home
d03$[5, 0]$espresso grind size
d04$[4, 1]$a warm drink before bed
d05$[3, 1]$milk for a kitten
d06$[2, 3]$feeding a rescue kitten
d07$[1, 4]$walking a dog in the rain
d08$[1, 5]$dog leash training
d09$[0, 4]$horse pasture rotation
d10$[2, 2]$the farm cafe menu
d11$[6, 1]$iced tea, two ways
d12$[9, 9]$quarterly revenue by region

Three of them are the right answers: d01, d02 and d11 all tell you how to make tea. I labeled that set by reading the twelve texts, before running anything, and it does not change for the rest of the chapter. A relevant set chosen after seeing the output would let the output decide what counts as right.

Stage one: retrieve

Score all twelve by cosine to the query and keep the best five. This is chapter 8's nearest-neighbor search, unchanged.

$$\cos(\mathbf{q},\, \mathbf{d}) = \frac{\mathbf{q} \cdot \mathbf{d}}{\|\mathbf{q}\|\,\|\mathbf{d}\|}$$

In words: the cosine between the query $\mathbf{q}$ and a document $\mathbf{d}$ is their dot product divided by both of their lengths. Dividing by the lengths throws the lengths away, so the ranking is about heading alone.

The twelve cosines, best first: d02 1.000, d11 0.999, d04 0.999, d05 0.992, d01 0.981, d03 0.981, d10 0.832, d12 0.832, d06 0.707, d07 0.428, d08 0.385, d09 0.196. The cut at five keeps d02, d11, d04, d05 and d01.

On the left, a grid with the twelve documents plotted as dots and labeled d01 through d12. A mint arrow runs from the origin out to the query. The five retrieved documents are ringed in mint; d11 is drawn in amber with a cross through it. On the right, a column headed 'cosine to the query' listing all twelve in order with their scores, d02 at 1.000 down to d09 at 0.196. The d11 row is struck through in amber and annotated 'dropped at dedupe'; d02 and d01 are annotated 'relevant'. A dashed line after the fifth row is labeled 'the cut at five'.
The ranked list, and the one that does not survive the next station. Notice where d12, a revenue report, lands: 0.832, above two of the animal documents.

That d12 score is chapter 2 arriving with consequences. The revenue report sits at $[9, 9]$, far from everything, and cosine ignores how far. It has the same heading as d10, the cafe menu, so it gets the same score. A system that ranked by distance instead would put it last.

Stage two: dedupe, and what it cost

Near-duplicate documents waste the user's attention and the reranker's budget, so most pipelines drop any candidate that is nearly identical to a candidate already kept. Walk the five in rank order and drop anything whose cosine to an already-kept document is 0.999 or higher.

Two pairs clear that bar. cos(d02, d11) is 0.999 and cos(d02, d04) is 0.999. Walking in order keeps d02, then drops d11, then keeps d04 because d04 was never compared against a keeper that scored high enough, then keeps d05 and d01. Four survive.

d11 is iced tea, two ways. It is not a duplicate of anything. It is one of the three documents I labeled relevant before the run, and the threshold killed it.

Stage three: rerank

The four survivors go to a scorer that reads the query and the document together, which chapter 14 calls a cross-encoder. Ours is a toy with two terms: start at three, subtract a sixth of the gap in each coordinate, and add one if the text says "tea".

The scores come out d02 4.000, d01 3.667, d04 2.833, d05 2.667. The order arriving was d02, d04, d05, d01. The order leaving is d02, d01, d04, d05.

Two ranked columns. The left column, headed 'ranked by the bi-encoder', lists 1 d02 at 1.000 brewing loose-leaf tea at home, 2 d04 at 0.999 a warm drink before bed, 3 d05 at 0.992 milk for a kitten, 4 d01 at 0.981 how to brew loose leaf tea. The right column, headed 're-ranked by the cross-encoder', lists 1 d02 at 4.000, 2 d01 at 3.667, 3 d04 at 2.833, 4 d05 at 2.667, with the moved rows in amber. Three curved amber arrows join each moved document to its new position. A line beneath reads '3 of the 4 positions changed; d01 passed 2 documents'.
Three of four positions changed. d01 climbs from fourth to second, passing two documents that the cosine had ranked above it.

d01 is how to brew loose leaf tea, and it was fourth of four on the cosine ranking. It says "tea" and it is close to the query, so the second scorer, which gets to look at both texts at once, puts it second. That is the whole case for reranking, and chapter 14 prices it.

Stage four: score the run

Two numbers score a ranked list against the relevant set, and they differ in their denominators, which is the whole reason they disagree.

$$\text{recall@}k = \frac{|R \cap \text{top-}k|}{|R|} \qquad\qquad \text{precision@}k = \frac{|R \cap \text{top-}k|}{k}$$

In words: both count how many of the relevant documents $R$ appear in the first $k$ results. Recall divides that count by how many relevant documents exist, so it asks what share of the right answers you found. Precision divides it by $k$, so it asks what share of what you returned was right. Different denominators, different questions.

Both scores on the retriever's ranked list, at four depths. The relevant set is d01, d02 and d11.
$k$Relevant foundrecall@$k$precision@$k$
1d020.3331.000
3d02, d110.6670.667
5d02, d11, d011.0000.600
10d02, d11, d011.0000.300

In the third row, recall@5 is 1.000: the retriever found all three. The pipeline handed back d02, d01, d04 and d05, which contains two of the three, so the pipeline's recall is 0.667. The two numbers are measured at different points on the line.

Every station, what reaches it, what leaves, and what it is able to remove.
StationInOutWhat it can lose
Embed the query1212Nothing
Retrieve top five125Seven documents, by the depth you chose
Dedupe54One document, by the threshold you chose
Rerank44Nothing; it reorders
Score the run44Nothing; it measures
Cluster1212Nothing; it groups
Flag anomalies1211One document, if you act on the flag
Watch drift1212Nothing; it compares

Stage five: cluster

Clustering is not part of answering the query. It is how you find out what is in the collection you are serving. Run chapter 8's k-means on all twelve documents with $k = 3$, starting the centers at $[6, 0]$, $[1, 5]$ and $[2, 2]$.

The within-cluster sum of squares, which is the total of every document's squared distance to its own center, falls at each pass: 76.417, then 70.417, then 20.333, then 16.167, and the fourth pass is where it stops moving. The centers settle at $[4.833, 0.667]$, $[9, 9]$ and $[1.2, 3.6]$.

The groups are the six drink documents, the five animal-and-farm documents, and d12 by itself. One cluster is the revenue report and nothing else. That is the outlier showing up as a group of one, which is a useful thing for a cluster report to tell you and a confusing thing if you expected three topics.

Press Step four times and the readout walks down the same four numbers the paragraph above lists. Then set the clusters to two and watch the revenue report get folded into a real group, which is the same document looking like a member instead of an event.

Stage six: watch it for anomalies

An anomaly here means a document far from the middle of the collection. Take the centroid of all twelve, which sits at $[3.667, 2.583]$, measure every document's distance to it, and you get a mean of 3.083 and a standard deviation of 1.769.

Flag anything past the mean plus two standard deviations, which is 6.621. Two is a convention, not a law, and twelve points do not earn a bell curve; what makes the rule safe on this corpus is the gap that follows. Exactly one document clears it: d12 at 8.344. The next-highest is d09 at 3.931, so there is a wide gap between the flagged document and everything else, and on this corpus the monitor is right. The revenue report does not belong in a recipe collection.

The threshold is also a maintenance job rather than a setting. It was computed from this month's twelve documents. Add a thousand documents about revenue and the mean moves out, the standard deviation grows, and 6.621 stops flagging anything.

Stage seven: watch it for drift

Re-embed the corpus next month. Every document shifts by $[0.3, -0.2]$, which is the kind of small global movement a retrained model produces, and one document genuinely moves: d06, feeding a rescue kitten, ends up at $[2, 5]$.

The corpus centroid goes from $[3.667, 2.583]$ to $[3.942, 2.567]$. The cosine between the old centroid and the new one is 0.999 and the gap between them is 0.276. A monitor watching the corpus-level number reports that nothing happened.

Now measure each document against its own old position. The four lowest cosines are d06 at 0.979, d10 at 0.993, d07 at 0.996 and d05 at 0.996. The event is right there, and it has a name attached to it.

The corpus average hid the one document that moved. Watch both numbers: the corpus-level one tells you whether the whole space moved, and the per-document one tells you which items moved inside it. Chapter 8 introduced drift on a single word; this is what it looks like when you have to run it as a monitor on a collection.

Stage eight: the budget

One query, end to end, on a corpus of a million documents rather than twelve, costs this. The stage counts stay the same; only the search step changes, because a million documents need chapter 8's approximate index.

$$t_{\text{query}} = t_{\text{encode}} + t_{\text{search}} + t_{\text{dedupe}} + t_{\text{rerank}} + t_{\text{assemble}}$$

In words: the time for one query is the sum of the five stages, and nothing overlaps, so the total is an addition rather than a maximum. Encoding runs once, the search runs once, and the rerank cost is the number of survivors multiplied by the cost of scoring one pair.

Encoding the query takes 8.000 ms, or 19.465 percent. The approximate search over 1,000,000 documents takes 12.000 ms, or 29.197 percent. Deduping five candidates takes 0.100 ms, or 0.243 percent. Reranking four survivors takes 20.000 ms, or 48.662 percent. Assembling the answer takes 1.000 ms, or 2.433 percent. The total is 41.100 ms.

One horizontal bar divided into five segments, each a different palette color. From left: encode the query, 19.465 percent, 8.000 ms; approximate search, 29.197 percent, 12.000 ms; a sliver for dedupe labeled by a leader line as 0.100 ms and 0.243 percent; rerank, 48.662 percent, 20.000 ms; and a narrow assemble segment labeled by a leader line as 1.000 ms and 2.433 percent. A line beneath reads 'one query, end to end: 41.100 ms'.
One station is half the bill. The rerank is 48.662 percent of the query, and it is scoring four documents.

So the first trade you would make is the rerank depth, because it is the only line that scales with a number you chose. Halve the survivors and you halve the largest segment. Chapter 14 works out what you give up: the rerank is the stage that fixed the ordering, and d01 was the document it rescued.

What this looks like at ten million

Nothing on this page changes shape when the corpus grows. The retrieve stage stops scanning and walks chapter 8's layered graph instead, which returns most of the true neighbors rather than all of them, so the recall you lose at stage one is now a setting in the index as well as a depth you chose. Storage is chapter 14's table: at 384 dimensions a million documents is 1.536 GB in float32 and 0.768 GB in float16, so ten million is ten times that, and it has to sit in memory for the search to be fast. The rerank stays at 5.000 ms per pair no matter how large the corpus gets, because it only ever sees the survivors. That is why the industry shape is a cheap wide first stage and an expensive narrow second one, and it is the same five lines of budget with bigger numbers in two of them.

Where you'll meet this

What you now know

  • A pipeline is a sorting line: a station can reject a document or reorder the survivors, and no station can recover something an earlier one dropped.
  • Ranking twelve documents by cosine puts a revenue report at 0.832, level with a cafe menu, because cosine compares headings and throws the lengths away.
  • A dedupe threshold of 0.999 removes d11, a relevant document, which is what a fixed cutoff on a score does when the score was not built to answer the question you asked it.
  • Reranking with a scorer that reads both texts changed three of the four positions and lifted d01 past two documents the cosine had ranked above it.
  • recall@5 on the retriever's list was 1.000 while the finished pipeline returned 0.667 of the relevant set, so a pipeline's recall is not its retriever's recall.
  • A corpus-level drift cosine of 0.999 said nothing had happened in a month when d06 had moved to 0.979, so the average hides the event the monitor exists to find.
  • One query costs 41.100 ms across five stages and the rerank is 48.662 percent of it, so the first thing you trade is how many survivors you rerank.

Where we're headed

Every number in this book so far has been one I could compute in front of you, which was the point. The questions that are actually open in this field cannot be settled that way, and some of them are open because nobody has run the right experiment yet. The last chapter names those questions in the terms the rest of the book has earned, and then hands you three experiments small enough to run on a laptop tonight, each with the control that turns a picture into evidence.