BM25 + Vector Search with RRF: how I stopped guessing at retrieval quality

Every RAG demo has the same origin story: embed everything, search by cosine similarity, show one magical answer, declare victory. Then real queries arrive — full of module codes, error strings and acronyms — and the embedding model, which has never cared about characters, returns beautifully semantic results that contain none of the words anyone typed.

This is the story of rebuilding retrieval in ragkit around two convictions: exact match matters, and retrieval quality is measurable.

Vector search is not a retrieval strategy on its own#

Embeddings compress meaning, and compression is lossy in exactly the places people search:

• Rare tokens vanish. "COMP0016" and "COMP0017" are neighbours in embedding space — often closer to each other than to their own paraphrases.
• Negations blur. "pass" and "fail" can sit uncomfortably close depending on the corpus.
• Freshness leaks. Anything after the embedding model's training cutoff is closest to whatever it resembles, not what it is.

Meanwhile BM25 — a lexical scorer from the 1990s — nails all three, and falls apart on the one thing embeddings own: paraphrase. Ask for "exam timetabling" and BM25 shrugs at documents saying "assessment scheduling".

The failure modes are almost perfectly complementary. So: run both, fuse the results.

Reciprocal Rank Fusion in fifteen lines#

RRF combines rankings instead of scores, which is exactly right here — BM25 returns unbounded log-scores, vectors return cosines, and no normalisation makes them safely comparable. Ranks, however, are universal:

def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60):
    """rankings: one list of doc ids per retriever, best first."""
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda item: item[1], reverse=True)

Each retriever contributes 1 / (k + rank) per document. A document ranked #1 by one retriever but absent from the other beats a document ranked #4 by both. That's the behaviour you want: agreement is evidence, but so is a strong vote of confidence.

Tuning notes from my notebooks:

• k = 60 is a fine default, but lower values (20–40) helped on small corpora where the top result really is special.
• Candidate depth mattered more than k. Each retriever contributing its top 50 before fusion beat top 10 fusion decisively.
• Don't overthink weights between retrievers. Equal weighting, then reranking, outperformed every hand-tuned weighting I tried.

If it isn't measured, it's vibes#

The evaluation harness is embarrassingly small: ~120 queries with labelled relevant documents, Recall@K and Precision@K computed per retriever. It changed the project's physics:

Numbers like these do something a demo can't: they convert every future argument ("should we raise chunk size?", "is this reranker worth the latency?") into an experiment with a scoreboard. The eval suite runs in CI, and the build fails if Recall@5 regresses past a threshold.

What actually moved the needle#

Ranked by impact, from my experiment log:

1. Hybrid + fusion — the single biggest jump, as the table shows.
2. Candidate depth before fusion — 50 per retriever, not 10.
3. Chunking by structure — academic PDFs chunked at headings beat token-count chunking, especially for tables.
4. A lightweight reranker — worth it for the top 20 candidates; not worth it deeper.

The uncomfortable truth#

Since adding evaluation, I trust retrieval systems less and understand them more. I can now tell you exactly which queries my engine fails and why — identifier queries with typos, multi-hop questions, anything whose answer lives in a table.

That's a much better place to build from than "it seems to work". If you take one thing from this post: build the tiny eval set first. Quality you can measure is quality you can ship.