I built a search engine over our issue tracker. About 100,000 tickets. My colleagues use it before they start debugging something, to check whether one of us already fixed the same thing last year.

I gave it two halves, because they catch different things:

  • BM25 matches exact words. This is the half that finds the ticket containing your error string, your version number, your component name.
  • Vector KNN matches meaning. This is the half that finds the ticket describing your problem in completely different words.

Both run on every search. I merge the two rankings with reciprocal rank fusion (RRF) and show one list. Standard recipe, and it works well.

Then I went through my search code line by line, and found that on the busiest path BM25 was returning zero results. Not fewer results. Zero, every time.

So half the system was switched off. The half that matches exact strings, the half you rely on when you paste in an error code, had not been contributing anything.

And nothing told me. The results page was full. The scores looked normal. No error, no empty state, no alert.

The setup

In production it runs on an A100 with 80 GB of VRAM. I develop it on a laptop with an RTX A1000 and 4 GB, and I wanted the whole thing to run there too. That tighter machine is what shaped my model choices.

  • Gemma-4 E2B-it, 4-bit nf4 quantised with bitsandbytes. It structures tickets and reads image attachments. Quantising it keeps it inside 4 GB, and on the A100 it leaves room for other services sharing the card.
  • multilingual-e5-large for embeddings, 1024 dimensions
  • Elasticsearch 9.x for both the keyword index and the vectors

When a ticket comes in, the model splits it into fields: problem, root cause, solution, keywords. When someone searches, BM25 and vector KNN both run, and RRF merges what they return.

One function, two very different queries

I have one search function. Two places call it.

# someone types into the search box
query_text = f"{request.title} {request.description}"

# someone uploads a ticket, and the model summarises it first
query_text = summary["embedding_text"]

The first one is a handful of words. The second is a whole model-written summary, and those are long. The median is 866 tokens.

Both go through the same BM25 query, with the same threshold in it:

"minimum_should_match": "30%"

Now, why 30%? Because I wrote that line while I was thinking about the first caller. For “login fails after update” it is exactly right: six words, at least two must match. It is also the value you see in most Elasticsearch examples, next to the same title^3, problem^2 field boosts I had copied along with it. For a search box it is a sensible default and I had no reason to look at it again.

Then the second caller reused the same function with a completely different shape of query, and nobody adjusted the threshold, because the threshold was not visible from there.

For an 866-token query, "30%" matches nothing. That summary comes out at roughly 300 unique terms once the analyser has finished with it, and minimum_should_match counts those terms. So a document had to share about 90 of them before Elasticsearch would even consider it.

BM25 minimum_should_match: 30% 0 results vector KNN multilingual-e5-large RRF fusion looks completely normal

Nothing shares 90 terms. So BM25 returns an empty list. RRF then merges one full ranking with one empty one. What comes out is plain vector search. It is still called hybrid.

Why this is worth knowing

The threshold is not the interesting part. What matters is why this can run for months without anyone noticing.

Vector search on its own is not bad search. It is decent search. So my results stayed useful enough that nothing felt wrong. They were just missing the ticket that named the exact error. And nobody can tell you about a result they never saw.

A hybrid system with one dead leg looks the same as a healthy one. Same shape of output, same speed, same green dashboards.

If a component returns junk, you notice straight away. If it returns nothing, you never do. Its weight quietly moves to whatever else is in the merge, and the merge carries on working.

The fix

I made the threshold depend on how long the query is:

def _min_should_match(query_text: str) -> str:
    term_count = len(query_text.split())
    if term_count <= 15:
        return "30%"      # short query, the normal default is fine
    if term_count <= 60:
        return "2<-25%"   # 2 words or fewer: match all. Longer: match 25%
    return "4"            # very long, use a fixed count not a ratio

The last line matters most. If you keep using a percentage, longer queries need more and more matching terms. A 2,000-token query at 25% still needs hundreds of them. That is the same bug again with bigger numbers. Past a certain length you have to drop the ratio for a fixed floor. Four solid term matches in a very long query is already a strong signal.

Elasticsearch supports the "2<-25%" form directly. It means: with 2 terms or fewer, match all of them; above that, match 25%. It is in the docs, and I had never noticed it.

How to check your own

The real lesson was not the threshold. It was that nobody was watching how many results each retriever returned.

You asked each retriever for k results. Warn whenever one of them comes back with noticeably less than that:

k = 50  # what you asked each retriever for

for name, hits in (("bm25", bm25_hits), ("knn", knn_hits)):
    if len(hits) < k * 0.5:
        logger.warning("hybrid degraded: %s returned %d/%d", name, len(hits), k)

Do not just check for zero. Asking for 50 and getting 2 is already a collapsed hybrid, and neither case shows up in the output.

In production, export the per-retriever hit counts as a metric rather than a log line. A warning that fires on every query gets ignored within a week.

The other thing worth doing takes longer. Go through the tuned numbers in your retrieval code, and for each one write down which caller you had in mind when you picked it. Then list everything that calls it now. Anywhere those two lists disagree is a number that has quietly stopped meaning what you meant.

Mine was a threshold. It could just as easily be a top_k, a chunk size, or a similarity cutoff.