The situation

Someone searches my issue tracker for a bug they are stuck on. The top result has a different title, about a different component, and does not obviously match anything they typed.

It might be an excellent match. The vector side may have seen that it is the same story underneath, described in different words. That is exactly what vector search is for.

But the person reading it cannot tell. All they see is a result that looks wrong sitting in first place. So they decide the search is broken and go back to asking a colleague.

This is a trust problem, and it is separate from ranking quality. I could improve the ranking forever and not fix it. A result nobody believes is worth about as much as a result that never came back.

The task

The system had to say why each result ranked where it did. Three constraints on that:

  1. It has to be fast. Search is used mid-debugging. If it stops feeling instant, people go back to asking a colleague and the whole thing is dead.
  2. It has to be true. Not plausible. True. The explanation is being used to decide whether to trust a surprising result, which is the one case where a wrong explanation does real damage.
  3. The number next to it has to agree with it. Showing “94% match” on a result ranked below one showing 91% undoes any amount of good explanation.

The action

Why I did not ask the model

The obvious move in 2026 is to hand the query and the document to an LLM and ask why they match. I already had a quantised model loaded. It would have taken an afternoon.

The speed constraint rules it out on its own. An LLM call in the search path adds seconds to every query, and the GPU is shared with other services.

But the second reason is the one that actually decided it. A model asked to explain a match will always write a convincing explanation, whether or not it is the real one. Give it a query and a document and it will find a connection between them. That is what it is good at.

The question I need answered is not “how are these two texts related”. It is “why did this ranking function put this document first”. Those are different questions, and a model only ever answers the first one.

An explanation that sounds right but describes reasoning the system never did is worse than no explanation at all. It teaches people to trust the system for reasons that are not the reasons. And it fails hardest on surprising results, which are the only ones anybody checks.

Building the explanation out of the ranking

By the time the results are ready, the search has already produced everything an honest explanation needs:

  • which fields BM25 matched, and where
  • which query keywords overlap the document’s keywords
  • whether the document was found by BM25, by the vector legs, or by both
  • how much each leg contributed to the final score

So the explainer reads those signals and writes a sentence from them. No model call. No extra latency. And nothing it says can be false, because every claim traces back to a number the ranker actually computed.

One detail mattered more than I expected: stopwords. In an issue tracker, words like error, issue, fail, problem and ticket appear in almost every document. “Matched on: error, issue” is completely true and completely useless. Worse than useless, because it makes the system look naive at the exact moment it is trying to earn trust. So those words are filtered out of the explanation, even though they still count toward the score. What is left is the vocabulary that actually separates this ticket from the other 100,000.

Making the number agree with the order

The obvious way to show a match percentage is to take the cosine similarity, multiply by 100, and print it.

That fails twice. First, e5 embeddings put almost all real cosine values in a narrow band, roughly 0.7 to 1.0. Every result comes out between 87% and 100%, a couple of points apart. The number cannot discriminate, so it tells the reader nothing.

Second, and worse, it can disagree with the ranking. The order comes from the merged RRF score, not from cosine. So the top result can display a lower percentage than the one beneath it. A user who sees that concludes the search is broken, and they are right to.

So the percentage is now derived from the score that actually sets the order. The top result is 100%, everything else is relative to it. The number agrees with the ranking by construction. And a document found only by BM25, which has no vector score at all, no longer shows “0% match” while sitting near the top.

Weighting the legs by what they mean

The merge combines five rankings, and the weights are deliberately uneven:

RANKING WEIGHT WHY problem vector 1.5 same kind of text as the query bm25 1.0 exact terms, error strings root_cause vec 1.0 shared underlying cause solution vector 0.7 restates its problem, weakly legacy vector 0.5 not yet migrated one _msearch weighted RRF rank constant 60 one ranked list

These come from what the fields are, not from a sweep. Someone searching is describing a problem they have not solved. Their text looks a lot like a problem section and only a little like a solution. Equal weights would let a ticket rank high just because its fix uses the same words as your symptom. That is the confusing result this whole thing exists to prevent.

All five go out in one _msearch. Running them one after another would cost a round trip per field.

The result

Before this, I measured how often the displayed percentage disagreed with the order it was printed next to. Nine of thirteen searches, 69%. Within the top five, the median gap between two results was 2.2 points, so the number was not discriminating anything either.

Both of those are now structurally impossible rather than fixed. The first result is 100% because the scale is defined that way, and every other result is a fraction of the score that ordered it. There is no arrangement of inputs that puts them out of step.

The explanations cost no measurable latency, because they are string formatting over numbers the search already had.

And the part I did not anticipate: making the weights explicit made them arguable. The problem leg is at 1.5 because I can say out loud why it deserves to be, and someone can tell me I am wrong. That is a better position than having the behaviour fall out of an implementation detail nobody ever looked at.

What I would take from this

If you are adding explanations to a retrieval system, the useful question is not “which model should write them”. It is “what does my ranker already know that it is throwing away”.

Almost always the answer is: most of it. The per-field matches, the per-leg contributions, which retriever found the document. All of that exists for a few milliseconds inside the search and then gets discarded so a single sorted list can be returned.

An explanation built out of those numbers cannot drift from what the system did. An explanation generated after the fact always can, and you will not be able to tell the difference by reading it. That is the whole problem.