Episode 6462 custom

When Adding a Reranker Hurts Search

NVIDIA benchmarked a cross-encoder upgrade that scored three points below its baseline, while other models surged fifteen.

Nov 5 19:40 Transcript

With

0:00

Episode Details

Episode ID: MWP-6664
Published: Nov 5
Duration: 19:40
Audio: Direct link
Pipeline: V5.26
TTS Engine: chatterbox-regular
Script Writing Agent: GLM-5.3
Topics: ai-inference ai-evaluation context-and-memory rag

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

Reranking is important and we've covered it before, as we have vector embeddings, but I want to understand how it all fits together in practice. So let's take two versions of the same search tool. It could be a RAG integration in an AI tool, or maybe even just the search box on a website. Version 1 has no reranking; version 2 does. What would the difference actually look like from the user's perspective, and what are the real performance differences between the two?

Two-stage retrieval architectures promise an easy quality boost by pairing fast bi-encoders with cross-encoder rerankers, but real-world results vary wildly. While benchmarks show gains of five to fifteen nDCG points, poor model choices can actually degrade performance below baseline, and strong initial retrieval setups can yield negligible improvements at massive latency costs. A reranker cannot promote documents never retrieved, making its performance strictly dependent on first-stage recall curves: the component only earns its keep when recall at a hundred candidates substantially exceeds recall at five. In production RAG pipelines, the reranker functions less as an accuracy silver bullet and more as an economic filter, spending cheap compute to winnow down inputs for expensive generator models.

Sources

What the research for this episode read before the script was written. Primary sources first.

Mentions

BAAI Beijing AI research institute behind BGE models
BGE Reranker v2 M3 BAAI workhorse cross-encoder reranker model
BM25 Classic lexical keyword retrieval algorithm
llama 3.1 8b Meta open-weight base model
Mistral AI model provider
ms-marco MiniLM L12 Small cross-encoder reranker model
NeMo Retriever NVIDIA's retrieval and reranking toolkit
NVIDIA GPU and AI hardware company
Qdrant High-performance vector search engine
SciFact Scientific abstract retrieval benchmark dataset

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6462: When Adding a Reranker Hurts Search

The standard story on rerankers is that they're the cheap upgrade every search system should have. Bolt one on, watch the right answers float to the top. That story is half right, and the half it gets wrong costs people months.

Which is exactly the trap Daniel is poking at this week.

He's saying: we've done embeddings, we've done reranking, fine, but separately. Take one search tool, a RAG pipeline or even just a website's search box. Version one, no reranker. Version two, reranker bolted on. What does the user actually see differently, and what are the real numbers between them. And the honest answer is the numbers disagree with each other, wildly, and the disagreement is the interesting part.

I'm Herman Poppleberry, and I've been looking forward to this one, because the two-stage architecture is one of those things where the mechanics are elegant and the marketing around it is terrible.

Two claims I'll be checking as we go.

You check. I'll explain. The machinery first, because you can't judge the numbers without it. Stage one is a bi-encoder. It takes the query and turns it into a vector, takes each document and turns it into a vector, and compares vectors with cosine similarity. The trick that makes it fast is that all the document vectors were computed once, at index time, before anybody ever asked anything. A query comes in, you embed just the query, and you're doing a nearest-neighbour lookup against precomputed vectors. Under a millisecond in the typical setup.

So the documents never meet the query. They meet the query's shadow.

That's a nice way to put it, actually. The document only ever sees a compressed summary of the query, compared against its own compressed summary. Stage two, the cross-encoder, is where they finally meet. You concatenate the query and the document into a single input and run them through the model together, so self-attention is comparing query tokens against document tokens, word by word. The BAAI model card has the crispest one-line version I know: reranker uses question and document as input and directly outputs similarity instead of embedding.

The price of the meeting is that you can't precompute any of it.

Right, that's the whole trade. Every query-document pair is a fresh forward pass through the model. If stage one retrieves a hundred candidates, that's a hundred fresh passes, per query, forever. The bi-encoder did that work once and amortized it across every query you'll ever serve.

So the practical question is what the user gets for the per-query bill. Daniel asked exactly that, so let's run his two versions.

Let's do version one first, and I want to steal Qdrant's own tutorial example, because it's a good one. The query is, what is the purpose of feature scaling in machine learning. The corpus contains documents that use the words feature scaling in other senses. And this is the important part of version one: the user doesn't get an empty results page. They get a full, confident-looking list, where the top slots are dominated by lexical and semantic near-misses. Documents that share vocabulary with the question without answering it.

The plausible wrong thing at the top. Which is worse than nothing, because nothing tells you to keep looking.

In a website search box you'd call it a bad search engine and retype the query. In a RAG pipeline it's nastier, because the system doesn't show the user the list. It takes the top five chunks and hands them to the language model, which will happily synthesize an answer from whatever it was given. The right chunk sitting at rank seven of ten is invisible to it. NVIDIA's writeup on this is blunt about it: reranking lets you pass fewer chunks with higher precision, because the model at the end only ever sees what you chose to give it.

And version two?

Version two, the cross-encoder reads the actual question against each near-miss and goes, no, this one's about scaling in the metallurgy sense, this one's about feature scaling in ML, and the correct document moves to position one. The user-visible difference isn't results versus no results. It's plausible wrong thing at the top versus the right thing at the top.

There's a second difference nobody advertises. Stability.

Go on.

The bi-encoder scores each document in isolation, so two near-duplicate documents get near-identical scores, and which one lands above the other is basically coin-flip territory. Rerun the same search, same order isn't guaranteed. The reranker sees the query next to both and can actually separate them.

That's real, and it's one of the cases where reranking earns its keep honestly. Hard queries where the first stage can't separate near-duplicates. Okay, now the numbers, because this is where Daniel's question gets uncomfortable.

I was wondering when we'd get to the part where the literature can't agree.

Start optimistic. The mid-year production survey that gets cited everywhere puts the typical lift at five to fifteen nDCG at ten points across the standard benchmarks, and its latency table has the bi-encoder baseline under one millisecond per query against fifty to five hundred milliseconds for reranking the top hundred. So: meaningful quality jump, for a real but bounded latency bill. That's the version of the story you'll read in most vendor posts.

Five to fifteen points is the kind of number that sells a component.

It is. Now the benchmark paper. NVIDIA ran a question-answering retrieval task with a decent embedding model as the baseline, nDCG at ten of about point six one, then stacked different rerankers on top of it. The best one, a Mistral four-billion reranker, took it to about point seven five. Fourteen and a half points. There's your optimistic case, confirmed. But the worst reranker in the same table, ms-marco MiniLM L twelve, scored point five eight. Below the baseline. Adding the component made the system measurably worse.

Three points worse for adding a part.

A three-point regression from an upgrade. Same architecture, same corpus, same baseline. The spread from best to worst reranker is entirely a model-choice question, and most of the blog posts about reranking don't mention that there's a downside tail at all.

Then there's the other end of the spectrum. Hurrell's reranker tax piece from the summer.

Eoin Hurrell, July. He took BM25 over SciFact, a few thousand scientific abstracts, three hundred queries, and reranked it with a small cross-encoder. BM25 alone: point six six two nDCG at ten, at about a tenth of a millisecond. Rerank at twenty candidates: point six seven one, at two hundred twenty-five milliseconds.

Two thousand times the latency.

For a gain of point zero zero nine that doesn't escape the confidence interval. Statistically indistinguishable from doing nothing, at two thousand times the cost.

So the field's published numbers run from plus fifteen to zero, and the measured ones run to minus three. That's not a spread, that's a disagreement about whether the thing works.

And both headline numbers are correct. That's the part I want to sit on, because the reconciliation is the single most important thing to understand about rerankers, and almost nobody leads with it.

Say it, then.

A reranker reorders what the first stage hands it. It cannot promote a document the retriever never returned. So the reranked score is hard-capped by first-stage recall at whatever k you fed it. Hurrell's own figure on SciFact: recall at five is about point seven two, at ten point seven seven, at twenty point eight two. Only at the top of that climb does the reranker's lift appear. At five and ten candidates, moving essentially zero.

Wait. So in his experiment the reranker looked useless because the first stage had already found the answer inside the top twenty, and the reranker's whole job is rescuing answers that are further down.

That's it exactly. His corpus was one where BM25 is unusually strong. The five-to-fifteen-point numbers come from benchmark suites where the first stage is a dense retriever with high recall at a hundred candidates, and there's a lot of room between recall at a hundred and what the user sees at the top five. Same technique, opposite verdicts, because the foundation differed. There's a rule of thumb in the production writeup that compresses the whole diagnostic: the reranker pays off when recall at a hundred is materially above recall at five on your eval set. That gap is the reranker's entire job description.

So the first question isn't which reranker. It's what your recall curve looks like.

Diagnose the first stage before you add the smart layer. Hurrell says something close to that verbatim, that the decision gets made as a fix for feelings of poor quality. Add the clever part without measuring the foundation.

Which is a very human move. The search feels bad, the fix must be the missing clever part.

Speaking of which, this is where I check the call sheet. Franklin, you're on.
Franklin: Sorry, I've been nodding along for weeks now and I'm fairly sure I've missed step one entirely. Is reranking just, like, when the search box puts the obvious answer first instead of the one with all the right words in it, the way Google guesses what I meant when I typed the wrong thing?

Close enough. Think of it like a librarian who first grabs every book with the right words on the spine, then actually opens the ones that look promising before handing you one.

Generous of you, after all that.

Back to it, then. The recall ceiling reframes everything, because it turns the reranker from an accuracy feature into something more like a cost optimization.

That's the NVIDIA argument, isn't it.

It is, and it's the better argument, honestly. Their pitch isn't reranking finds better documents, it's arithmetic. A Llama three point one eight-billion model costs roughly seventy-five times more to process five chunks and generate an answer than the NeMo retriever reranker does per token. So the reranker is seventy-five times cheaper than the generator. Which means you can afford to rerank a wide candidate set, a hundred chunks, and pass fewer chunks to the language model, and the total bill goes down while quality goes up. They report about twenty-one and a half percent cost savings from that rebalancing alone.

Spend the cheap compute to shrink the input to the expensive compute.

That's the entire game. The reranker's real job in a production RAG system is being a cheap bouncer that decides what the expensive model gets to see.

Let's put numbers on the bouncer's salary.

Per-pair, on a single L40S GPU: the little MiniLM twelve reranker is around four milliseconds a pair, so four hundred milliseconds to do a hundred. The workhorse, BGE Reranker v2 M3, about twelve milliseconds a pair, one point two seconds for a hundred. The big Qwen3 eight-billion reranker, around thirty-eight a pair, nearly four seconds for a hundred candidates. And infrastructure moves the number as much as model size: one independent harness measured the same BGE model self-hosted at around two point four seconds average against Cohere's hosted Rerank three point five at under four hundred milliseconds for the same job. Six times, from deployment choice alone.

And the hosted bouncer costs a tenth of a cent per search.

Cohere charges a thousandth of a dollar per search, one query with up to a hundred documents. LiteLLM lists the same at two dollars per thousand queries, as of this week. The self-hosting break-even lands somewhere around a million queries a day. Below roughly ten thousand a day, the API is the sensible default, no GPU operations to babysit.

So we've got a decision tree. When is the tax worth paying?

Hurrell's conditions are the ones I'd sign: high-stakes, low-volume retrieval. Hard queries where the first stage can't separate near-duplicates. Subtle ordering mattering enough to justify a listwise reranker. And you pay the tax three ways, latency, cost, and complexity, because now you're running and monitoring a second service that can fail independently of the first.

Which is the part that never makes it into the architecture diagram.

Never. One more wrinkle worth naming, because it's where this touches what Daniel actually builds. The adoption picture: a Q1 survey from a reranking vendor put about thirty-six percent of production RAG deployments at no reranker at all. Vendor survey, not audited, but it rings true. Most systems ship the bi-encoder and stop.

That number I believe, if only because I've used enough of the search boxes this industry produces.

Here's the second-order part, and I think it's the freshest thing in all of this. Hurrell draws the parallel to agents, and it's a sharp one. Point an AI agent at a weak search tool and you reproduce the reranker mistake one layer up. The agent searches, doesn't find what it needs, reformulates, searches again, loops, burns tokens, and looks productive the whole time. Every rephrased query still runs against the same index, the same embeddings, the same blind spots. His line is that reformulation lets the agent ask differently, not make the retriever return a document it has no path to.

The recall cap doesn't go away. It just hides under the loop.

And the loop costs more than the reranker would have. It's the same unmeasured upgrade, paid out in generator tokens instead of milliseconds.

Herman, before we go on. You've been building out your little retrieval test bench in the spare room.

It's not little, it's a proper eval harness, and yes, I got the recall curve first this time. Old lesson, learned the hard way.

It was ten. I widened it to fifty.

Fifty what?

Candidates. Before the reranker touches anything. On the forum I moderate. We had the search the way you're describing, Corn, where the confident wrong thing sits on top. I kept saying the reranker was broken. It wasn't broken. My candidate pool was ten and the good thread was sitting at position twenty-three, and no reranker on earth reaches down to twenty-three out of ten.

How do you know it was at twenty-three?

Because I score the top ten by hand. Every week, same queries, a set I fixed in two thousand eight. I print the results and I keep them in a binder, and I mark them. Yellow is answers the question, orange is uses the words, green is my own category. I compute my own score off those marks. It's not the standard formula. Mine weights thread actually answers the question above everything, which the standard one does not do, whatever the rest of you are doing with your discounting. Corn, you'd like the binder. You can hold it. I use three highlighters.

I have questions about the highlighters.

You can have questions. I don't explain the system, it works. The point is the spreadsheet doesn't lie. Same reranker, same forum, everything the same, and the moment I widened the pool from ten to fifty the right threads started arriving at the top. It was the pool, not the smart layer. Herman's been saying that for ten minutes in fancier words.

So you've been hand-computing an nDCG variant on a hobby forum since the late two-thousands.

I don't call it that. I call it the standing order. Twenty-three, if anyone wants the number.

He kept the number.

Of course he kept the number. That's... actually consistent with everything we've just said, Corn. His binder is a recall-at-k measurement with extra steps. He found the same ceiling, from the other side, with highlighters.

I want to see the green category.

Later. Where were we, the agent parallel. The one thing I'd flag from the whole discussion is that reframing. If reranked quality is capped by first-stage recall, and agents are now looping and reformulating on top of the same capped retrieval, the next real improvement isn't a better reranker at all.

It's a better first stage.

Or a different answer to where quality actually comes from. And there's a gap in the literature that surprised me: nobody seems to have run a true head-to-head A/B of the same production search tool with and without reranking, measured on user behavior. Click-through, task completion, session length. Every number we quoted today is offline retrieval metrics or latency and cost. The user-perspective half of Daniel's question is inferred from rank order, not measured.

Six thousand episodes in and the field still hasn't watched a real user type a real query into both versions.

That's the experiment I'd fund first.

Then let's leave it there for today. Thanks as always to our producer, Hilbert Flumingtop, holding down the desk and the binder.

If today's discussion changed how you think about search, the website has the full archive, my weird prompts dot com.

We'll be back soon. Until then, keep your candidate pool wide.

And measure the recall curve before you buy the smart layer. See you tomorrow.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error

When Adding a Reranker Hurts Search

Sources

Mentions

Downloads

You Might Also Like

#6462: When Adding a Reranker Hurts Search