ragkit

The problem#

Most RAG demos die on the second question. A vector index over chunked documents looks magical until someone searches for an exact identifier — a module code, an error string, a student ID — and the embeddings return "semantically similar" results that contain none of the characters the user typed.

That tension is the whole design brief for ragkit: an exact-match-friendly retriever for a UK university Q&A corpus, where queries mix natural language ("when do exams start?") with brittle identifiers ("COMP0016 assessment").

Why hybrid retrieval#

Lexical and semantic retrieval fail in opposite directions:

• BM25 nails exact terms and rare tokens, but misses paraphrases entirely.
• Vector search matches intent across paraphrases, but blurs rare tokens into nearest-neighbour soup.

A hybrid retriever runs both and fuses the rankings. The mid-2000s answer to that is Reciprocal Rank Fusion, and it still works suspiciously well.

Fusing with RRF#

RRF ignores raw scores — which is its superpower, because BM25 scores and cosine similarities aren't comparable anyway — and instead combines ranks:

def reciprocal_rank_fusion(
    rankings: list[list[str]],  # each: doc ids, best first
    k: int = 60,
) -> list[tuple[str, float]]:
    """Fuse multiple ranked lists into one. k dampens the influence of top ranks."""
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda item: item[1], reverse=True)

Two knobs matter in practice:

• k (default 60) controls how aggressively top-ranked documents dominate. Lower values make the fusion spikier.
• Candidate depth: how many results each retriever contributes before fusion. Retrieving 50 candidates per retriever and fusing down to 10 consistently beat any single-retriever baseline.

Measuring instead of guessing#

Retrieval quality is seductive to hand-wave. ragkit computes Recall@K and Precision@K against a labelled query set, both per-retriever and post-fusion:

The interesting number isn't that hybrid wins — it's where each retriever wins. Exporting per-query wins as a CSV made the failure modes concrete: BM25 dominates identifier queries, vectors dominate conceptual ones, and RRF takes the best of both.

CI as a quality gate#

A Makefile target runs the full eval suite and writes a report; a GitHub Actions workflow runs it on every push and fails the build if Recall@5 regresses beyond a small tolerance. Retrieval quality became a build artefact, not a vibe.

- name: Run retrieval eval
  run: make eval EVAL_THRESHOLD=0.75

What I'd do differently#

• Start with question-type routing earlier. A tiny classifier that sends identifier-looking queries straight to BM25 would cut latency and noise.
• Treat chunking as a first-class parameter. Academic PDFs break embedding quality at heading boundaries far more than at token limits.
• Log every production query (anonymised) from day one — nothing beats real query distributions for tuning.